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Logistics  Management  Institute 


Meeting  Customer  Service  Standards 
Under  Executive  Order  12862: 
NASA's  Space  Science  Grant  Process 


Executive  Summary 


Executive  Order  12862,  Setting  Customer  Service  Standards,  requires  that  agen¬ 
cies  take  customer  surveys  periodically  to  gauge  the  level  of  satisfaction  with 
government  services  and  to  improve  the  delivery  of  those  services,  as  necessary. 
The  National  Aeronautics  and  Space  Administration  (NASA)  identified  the  grant 
selection  process  used  by  its  Office  of  Space  Science  as  an  area  requiring  assess¬ 
ment  under  the  Executive  order.  That  process  has  two  customers:  scientists 
who  submit  proposals  in  response  to  NASA  Research  Announcements  and  their 
peers  who  hold  panel  meetings  to  review  the  merits  of  the  proposals.  The  Office 
of  Space  Science  asked  the  Logistics  Management  Institute  to  seek  the  opinions 
of  the  proposal  writers  and  peer  review  panelists  concerning  the  effectiveness 
and  efficiency  of  NASA's  science  grant  process  and  to  suggest  improvements. 
Our  survey  methods  were  approved  by  the  Office  of  Management  and  Budget. 

We  found  that,  on  average,  survey  respondents  are  moderately  satisfied 
with  the  overall  science  grant  process  and  with  the  resultant  quality  of  the  re¬ 
search.  Specifically,  the  peer  review  panelists  are  quite  satisfied  with  the 
support  —  facilities,  lodging,  and  supplies  —  that  NASA  provides  for  panel 
meetings.  They  also  are  reasonably  satisfied  with  the  time  that  NASA  allocates 
for  reviewing  the  proposals  before  the  panel  meetings  convene.  The  proposal 
writers  are  quite  satisfied  with  NASA's  guidelines  on  the  content  and  prepara¬ 
tion  of  the  proposals.  They  also  believe  that  three  months  is  an  adequate  period 
for  preparing  and  submitting  the  proposals. 

On  the  other  hand,  both  proposal  writers  and  peer  review  panelists  believe 
that  the  current  science  grant  system  is  less  effective  and  less  efficient  than  it  was 
just  a  few  years  ago.  Proposal  writers  said  that  they  must  write  five  or  more  pro¬ 
posals  in  the  hope  of  receiving  a  single  grant.  To  accommodate  the  increased 
number  of  proposals  being  written,  lead/secondary  peer  reviewers  indicated 
that  they  now  critique  an  average  of  seven  proposals  for  each  panel  meeting  —  a 
workload  that  has  reduced  the  effectiveness  of  their  proposal  reviews.  The  en¬ 
tire  grant  process  now  averages  more  than  13  months  from  the  time  NASA  calls 
for  research  proposals  in  a  NASA  Research  Announcement  (NRA)  to  the  time  it 
disburses  funds  to  those  awarded  grants. 

We  believe  that  the  Office  of  Space  Science  should  reduce  the  time  for  proc¬ 
essing  its  space  science  grants.  Survey  respondents  suggested  that  NASA  estab¬ 
lish  a  nine-month  time  standard  for  processing  grants.  The  nine-month  time 


standard  would  maintain  proposal  preparation  time  at  three  months,  reduce 
evaluation  feedback  and  notification  time  (now  more  than  six  months)  to  four 
months,  and  reduce  the  time  it  takes  to  disburse  funds  (now  more  than  four 
months)  to  less  than  two  months. 

To  meet  the  stricter  time  standards,  we  believe  that  the  Office  of  Space  Sci¬ 
ence  should  take  the  following  steps: 

♦  Revise  program  scope  in  NRAs  to  specify  research  priorities  and  associated  maxi¬ 
mum  funding  levels.  Because  the  current  program  scope  does  not  provide 
enough  information,  proposal  writers  prepare  and  submit  proposals  for  re¬ 
view  that  have  little  or  no  chance  of  receiving  funding.  By  including  more 
information  on  priorities  and  maximum  funding  levels  in  program  scope, 
NASA  could  significantly  reduce  proposal  and  review  workloads  without 
adversely  affecting  the  quality  of  its  grant  research  program. 

♦  Employ  independent  mail-in  reviews  before  peer  panel  reviews.  Peer  review  pan¬ 
elists  must  spend  too  much  time  on  proposals  that  are  obviously  unaccept¬ 
able  for  funding.  As  a  result,  they  have  insufficient  time  to  evaluate 
carefully  the  many  fine  proposals  that  are  submitted.  Peer  review  panelists 
could  spend  their  time  more  effectively  if  NASA  added  a  step  to  the  review 
process:  independent  mail-in  reviews  to  cull  the  proposals  that  are  unac¬ 
ceptable  for  funding.  If  NASA  adopted  such  an  approach,  peer  review  pan¬ 
elists  could  focus  their  attention  on  only  those  proposals  that  are  acceptable 
for  funding.  Moreover,  a  two-step  process  would  reduce  workloads  and 
raise  review  effectiveness  without  increasing  overall  processing  costs. 

♦  Consider  applying  information  technologies  to  parts  of  the  science  grant  process.  Of 
the  survey  respondents,  80  to  90  percent  approve  of  using  information  tech¬ 
nologies  for  preparing  proposals,  66  to  76  percent  approve  of  submitting 
proposals  via  information  technologies,  and  70  to  80  percent  approve  of  no¬ 
tifying  proposal  writers  of  evaluation  results.  NASA  should  further  review 
and  apply  computer  automation,  Internet,  and  electronic  data  interchange  to 
its  science  grant  process,  as  appropriate. 

Collectively,  these  and  other  actions  specified  in  this  report  can  enhance  the 
effectiveness  and  efficiency  of  the  science  grant  process.  Some  of  these  actions 
can  be  implemented  quickly,  but  others  may  take  considerable  time,  perhaps  un¬ 
til  the  end  of  FY96.  Consistent  with  Executive  Order  12862,  NASA  should  con¬ 
duct  additional  science  grant  customer  surveys  after  these  changes  have  been 
fully  implemented. 
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Chapter  1 

Introduction 


Executive  Order  12862,  Setting  Customer  Service  Standards,  directs  the 
executive  branch  to  take  customer  satisfaction  surveys  periodically  to  determine 
what  customers  think  of  the  government  services  they  receive  and  to  improve 
the  delivery  of  those  services,  as  necessary.  The  Executive  order,  which  is  shown 
in  its  entirety  in  Appendix  A,  defines  a  customer  as  an  external  "individual  or 
entity  who  is  directly  served  by  a  department  or  agency." 

In  addressing  the  requirements  of  the  President's  Executive  order,  the  Na¬ 
tional  Aeronautics  and  Space  Administration  (NASA)  identified  the  grant  selec¬ 
tion  process  directed  by  the  Office  of  Space  Science  as  an  important  area  that 
serves  external  customers.  Those  customers  are  proposal  writers  and  peer  re¬ 
view  panel  members.  Scientists  submit  proposals  in  response  to  periodic  NASA 
Research  Announcements  (NRAs),  and  their  peers  rate  the  worthiness  of  those 
proposals.  NASA  management  then  awards  grants  after  considering  the  panel 
proposal  ratings  in  the  context  of  NASA  missions,  budgets,  and  other  considera¬ 
tions.  To  comply  with  the  requirements  of  Executive  Order  12862,  NASA 
engaged  the  Logistics  Management  Institute  (LMI)  to  develop  customer  satisfac¬ 
tion  surveys  for  both  proposal  writers  and  peer  review  panelists  as  well  as  to 
conduct  those  surveys  and  analyze  the  results. 

The  following  subsections  briefly  describe  NASA's  science  grant  process  and 
discuss  our  approach  to  surveying  the  customers  of  that  process. 


Space  Science  Grant  Process 

The  NASA  science  grant  process  is  depicted  in  Figure  1-1.  The  process  be¬ 
gins  with  the  determination  of  research  needs  that  are  suitable  for  the  grant  pro¬ 
gram.  Those  research  needs  are  defined,  in  part,  by  Congress  and,  in  part,  by 
NASA  officials.  NASA  officials  also  obtain  input  from  various  science  advisory 
panels. 

To  inform  the  research  community  of  those  areas  in  which  NASA  has  special 
research  interests,  NASA  uses  the  NRA  —  a  form  of  "broad  agency  announce¬ 
ment"  described  in  6.102(d)(2)  and  35.016  of  the  Federal  Acquisition  Regulation. 
The  NRA  permits  the  competitive  selection  of  basic  research  projects  in  the  form 
of  grants. 


1-1 


Figure  1-1. 

Science  Grant  Process 


In  accordance  with  NRA  instructions,  proposal  writers  prepare  grant  pro¬ 
posals  for  conducting  research.  The  proposal  writers  not  only  discuss  the  con¬ 
tent  of  the  research,  but  also  provide  costing  and  scheduling  information  in  their 
proposals.  Proposal  writers  are  permitted  up  to  three  months  to  respond  to  an 
NRA. 

Peer  review  panels  meet  periodically  in  different  parts  of  the  country  to 
evaluate  proposal  writer's'  responses  to  the  NRAs.  Subpanels,  or  working 
groups,  are  formed  to  evaluate  the  scientific  worthiness  of  the  proposals.  Typi¬ 
cally,  a  primary  and  one  or  more  secondary  reviewers  are  assigned  to  each  pro¬ 
posal  for  documenting  its  strengths  and  weaknesses,  for  providing  merit  ratings, 
and  for  presenting  these  findings  to  the  entire  subpanel.  Some  panels  also  use 
independent  mail-in  reviews  to  supplement  their  views.  The  subpanel  then  de¬ 
velops  a  consensus  rating  for  the  proposal. 

A  full,  cross-discipline  panel  reviews  all  of  the  subpanel  results.  The  full 
panel  examines  each  evaluation  and  merit  rating,  normalizes  the  subpanel  pro¬ 
posal  ratings  for  differences  in  rating  systems,  and  judges  each  proposal  in  terms 
of  established  rating  categories.  The  established  rating  categories  are  as  follows: 


♦  Category  I  —  well-conceived  and  scientifically  and  technically  sound  investi¬ 
gations  that  are  pertinent  to  the  goals  of  the  program  and  the  announced  ob¬ 
jectives 

♦  Category  II — well-conceived  and  scientifically  or  technically  sound  investi¬ 
gations  that  are  recommended  for  acceptance,  but  at  a  lower  priority  than 
Category  I 
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♦  Category  III  —  scientifically  or  technically  sound  investigations  that  require 
further  development  and  may  be  reconsidered  at  a  later  time  for  the  same  or 
other  opportunities 

♦  Category  IV —  proposed  investigations  that  are  rejected  for  the  particular  op¬ 
portunity  under  consideration,  whatever  the  reason. 

These  rating  categories  represent  the  final  input  of  external  scientists  to  the  selec¬ 
tion  process.  The  time  standard  set  by  NASA  for  reviewing  and  providing  feed¬ 
back  on  the  evaluation  of  proposals  is  six  months. 

The  Program  Office  at  NASA  Headquarters  examines  the  peer  review  rat¬ 
ings  and  recommends  the  program  of  research  that  will  optimize  total  mission 
return  —  considering  scheduling,  engineering,  and  budgetary  constraints.  A 
Steering  Committee  reviews  the  recommendations  of  the  Program  Office,  the 
category  ratings  of  the  peer  review  panels,  general  compliance  with  procedures, 
and  adequacy  of  documentation.  That  Steering  Committee  then  prepares  an  as¬ 
sessment  for  the  appropriate  selection  official,  such  as  the  Associate  Administra¬ 
tor.  The  selection  official  notifies  the  proposal  writer  about  research  grant 
awards. 

Once  proposal  writers  have  been  awarded  grants,  NASA  provides  the  fund¬ 
ing.  It  has  no  time  standard  for  disbursing  the  funds  after  it  has  notified  a  pro¬ 
posal  writer  of  a  grant  award  for  conducting  the  proposed  research. 

The  researchers  are  responsible  for  communicating  their  progress  and  for  ac¬ 
counting  for  their  expenses.  NASA  reviews  these  communications  and  assesses 
grant  compliance  and  research  continuity.  The  researchers  also  are  responsible 
for  disseminating  the  research  results  at  scientific  conferences  and  in  research 
journals.  However,  some  of  NASA's  science  advisory  panels  may  request  brief¬ 
ings  of  research  results  for  some  major  grants. 


Study  Approach 

Government  Performance  Measures 

A  key  feature  of  the  Executive  order  is  the  establishment  of  customer  stan¬ 
dards  with  which  to  target  performance  improvements.  LMI  has  developed  a 
comprehensive  measurement  system  for  gauging  government  performance.1 
Specifically,  we  define  government  performance  in  terms  of  cost,  output  levels, 
and  quality: 

♦  Cost  reflects  operating  outlays,  appropriate  allocations  of  general  and  ad¬ 
ministrative  expenditures,  and  depreciation  charges  for  use  of  capital  items. 


^MI  Report  NS302R1,  National  Aeronautics  and  Space  Administration:  An  Approach  for 
Meeting  Customer  Standards  Under  Executive  Order  12862,  Lawrence  Schwartz  and  Brian 
Mansir,  May  1994. 
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♦  Output  levels  can  be  measured  by  the  number  of  units  produced  or  serv¬ 
iced. 

♦  Quality  includes  product  or  service  characteristics,  schedule,  and  timeliness. 

By  our  framework,  government  performance  can  be  improved  by  introduc¬ 
ing  new  processes  that  have  positive  effects  on  cost,  output  levels,  quality,  or  a 
combination  of  the  three.  Adopting  a  new  process  that  reduces  costs,  without 
degrading  quality  or  reducing  output  capability,  clearly  is  an  improvement  in 
government  performance.  Also,  adopting  a  new  process  that  enhances  quality, 
without  slowing  output  and  raising  costs,  improves  government  performance. 
In  addition,  adopting  a  new  process  that  expands  output  capability,  without 
hurting  quality  or  raising  costs,  is  a  benefit  to  performance.  Moreover,  processes 
that  simplify  operations  often  simultaneously  reduce  costs,  enhance  quality,  and 
increase  output  capabilities,  thereby,  yielding  major  improvements  in  govern¬ 
ment  performance.  Therefore,  NASA  should  evaluate  any  process  change  in 
terms  of  its  effects  on  cost,  output,  and  quality  —  total  performance. 

In  our  surveys  of  proposal  writers  and  peer  review  panelists,  we  focused  on 
measuring  the  components  of  quality.  We  also  covered  costs  and  outputs  to 
some  extent,  but  NASA  should  supplement  this  survey  information,  as  neces¬ 
sary,  to  complete  its  measurement  of  total  performance.  In  this  way,  NASA  will 
be  prepared  to  consider  changes  to  the  science  grant  process  in  terms  of  their  full 
effects  on  quality,  costs,  and  output. 


Key  Survey  Plans 

When  10  or  more  external  customers  participate  in  a  government  survey, 
the  Paperwork  Reduction  Act  requires  that  the  agency  submit  a  survey  plan  to  the 
Office  of  Management  and  Budget  (OMB)  for  approval.  Under  Executive  Order 
12862,  OMB  can  grant  a  generic  survey  clearance  to  an  agency  that  permits  it  to 
take  customer  satisfaction  surveys  without  prior  survey  approval.2  OMB  ap¬ 
proved  NASA's  generic  survey  clearance  request.  It  describes,  among  other 
things,  NASA's  approach  to  identifying  areas  of  improvement,  selecting  custom¬ 
ers  to  be  surveyed,  developing  survey  methods  and  collection  procedures,  mini¬ 
mizing  the  burden  on  survey  respondents,  and  reporting  survey  results. 
Appendix  B  shows  NASA's  generic  survey  clearance  request. 

The  OMB  approved  NASA's  generic  survey  clearance  request  for  one  year, 
through  Oct.  31,  1995  (see  Appendix  C).  NASA  may  renew  the  clearance  by 
keeping  OMB  informed  of  the  surveys  taken  during  the  year  and  of  plans  for  fu¬ 
ture  surveys. 

Our  approach  for  surveying  proposal  writers  and  peer  review  panelists 
complies  with  OMB's  requirements  as  well  as  follows  good  statistical  practices. 
First,  OMB  recommends  assessing  different  ways  to  contact  respondents.  We 


2  See  Statistical  Policy  Office,  Office  of  Management  and  Budget,  Executive  Office  of 
the  President,  Resource  Manual  for  Customer  Surveys,  October  1993. 
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dedded  to  use  mail  surveys.  In-person  interviews  are  considerably  more  costly 
for  obtaining  the  type  of  grant  information  required,  and  telephone  survey  inter¬ 
views  would  require  respondents  to  listen  intensely  for  too  long  a  period,  proba¬ 
bly  20  minutes,  jeopardizing  the  receipt  of  high-quality  data. 

Second,  OMB  requires  that  a  mail  questionnaire  be  answerable  within 
15  minutes.  This  OMB  time  limitation  means  that  the  survey  must  be  designed 
with  multiple  choice  or  close-ended  questions  —  rather  than  essay  or  open- 
ended  questions  —  and  yet  the  science  grant  process  must  be  comprehensively 
covered.  It  also  means  that  the  survey  should  be  pretested  (preferably  in  focus 
groups  of  less  than  15  participants)  to  determine  actual  time  burdens,  to  rephrase 
questions  that  are  difficult  to  comprehend,  to  delete  questions  that  are  inappro¬ 
priate,  and  to  add  questions  that  are  of  interest  to  the  survey  participants.3 

Third,  OMB  requires  that  everything  possible  be  done  to  minimize  survey 
nonresponses.  Nonresponses  bias  the  survey  results  to  the  extent  that  the 
opinions  of  the  nonrespondents  are  different  from  those  of  respondents.4 
Because  it  is  difficult  to  determine  if  opinions  differ,  it  is  prudent  to  adopt  a  se¬ 
ries  of  procedures  for  minimizing  survey  nonresponses  in  the  first  place.  Two 
procedures  are  particularly  noteworthy  for  mail  surveys:  small  survey  instru¬ 
ments  and  multiple  contacts.  Survey  research  reveals  that  a  bulky  survey  instru¬ 
ment  (for  example,  8%"x  11"  sheets  stapled  together)  invites  nonresponses;  to 
overcome  that  common  objection,  we  used  a  booklet.  Survey  research  also  re¬ 
veals  that  nonresponse  rates  tend  to  be  high  unless  would-be  respondents  are 
contacted  several  times;  to  ensure  high  response  rates,  we  planned  four  contacts.5 

Fourth,  while  OMB  permits  sampling  to  save  costs,  it  requires  that  the  sam¬ 
ple  be  planned  carefully  to  minimize  error  introduced  by  sampling.  To  bound 
the  sampling  error,  we  used  the  following  statistical  criterion:  select  a  sample 
size  that  provides  accurate  survey  answers  within  plus  or  minus  3  percentage 
points  at  the  95  percent  confidence  level.  We  selected  our  samples  from  the 
populations  of  proposal  writers  and  peer  review  panelists  accordingly.  Consid¬ 
ering  nonresponses,  we  accepted  a  margin  of  error  of  plus  or  minus  5  percentage 
points. 

Fifth,  OMB  stresses  the  importance  of  planning  for  the  analysis  of  the  survey 
data.  Because  quality  has  so  many  dimensions,  we  planned  to  use  the  survey 
data  to  develop  a  quality  index  that  would  enable  us  to  prioritize  improvement 
goals.  That  index  also  can  be  used  as  the  basis  for  tracking  customer  satisfaction 


3  The  pretest  is  part  of  the  total  design  method  developed  by  Don  A.  Dillman,  Mail 
and  Telephone  Surveys:  The  Total  Design  Method,  New  York:  John  Wiley  &  Sons,  1978. 

4  See  Office  of  Management  and  Budget,  The  Customer  Survey  Series:  How  to  Evaluate 
Alternative  Sample  Designs  for  Customer  Surveys,  March  15  - 16, 1994. 

5  Survey  research,  such  as  with  the  total  design  method,  has  shown  that  response 
rates  of  about  75  percent  have  often  been  achieved  with  multiple  contacts. 
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gains  resulting  from  the  implementation  of  those  quality  improvement  goals.6 
Our  analytical  approach  also  included  the  use  of  graphical  displays  to  summa¬ 
rize  responses  to  each  question,  identification  of  statistical  relationships  among 
the  answers  to  different  survey  questions,  and  determination  of  whether  re¬ 
sponses  to  a  question  are  universal  or  whether  they  are  dependent  on  the  back¬ 
grounds  of  the  survey  respondents. 


Organization  of  Report 

This  report  is  organized  for  different  types  of  readers.  NASA  officials  will 
be  most  interested  in  Chapter  2,  which  highlights  the  survey  results,  discusses 
the  implications  of  those  results,  and  suggests  targets  to  improve  the  science 
grant  process. 

Readers  interested  in  somewhat  more  detail  on  survey  methods,  results,  and 
their  implications  should  read  Chapters  3,  4,  and  5.  Chapter  3  discusses  the  de¬ 
sign  of  the  surveys  for  both  the  peer  review  panelists  and  the  proposal  writers,  as 
well  as  the  procedures  we  used  for  conducting  the  surveys.  Appendices  D  and  E 
contain  the  surveys,  and  Appendix  F  contains  the  post  cards  and  letters  sent  to 
would-be  respondents  to  minimize  nonresponses. 

Chapters  4  and  5  present  the  key  results  of  the  surveys,  along  with  sug¬ 
gested  improvement  targets  and  implementation  strategies  for  enhancing  cus¬ 
tomer  satisfaction  with  the  science  grant  process.  Appendices  G  through  J 
present  the  survey  results  in  detail.  Appendices  G  and  I  show  the  survey  ques¬ 
tions  and  the  distribution  of  responses  made  by  the  peer  review  panelists  and 
proposal  writers,  respectively,  and  Appendices  H  and  J  contain  comments  and 
suggestions  made  by  the  survey  respondents. 


6  See  National  Performance  Review,  Vice  President  A1  Gore,  Creating  A  Government 
That  Works  Better  &  Costs  Less,  Sept.  7, 1993.  The  quality  index  is  derived  by  taking  each 
quality  measure  of  customer  satisfaction  as  a  percentage  of  its  corresponding  customer 
standard.  The  result  is  a  series  of  customer  satisfaction-to-standard  percentages.  Each 
customer  satisfaction-to-standard  percentage  is  weighted  by  the  relative  degree  of  im¬ 
portance  that  customers  attach  to  it. 
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Chapter  2 


Survey  Highlights  and  Implications 


In  this  chapter,  we  outline  the  highlights  of  the  survey  results  for  both  the 
peer  review  panelists  and  proposal  writers.  We  also  discuss  the  implications  of 
those  survey  results  and  suggest  targets  for  improving  the  science  grant  process. 


Highlights 

Peer  review  panelists  are  fairly  satisfied  with  the  quality  of  the  research  that 
results  from  the  current  science  grant  process.  The  scores  averaged  3.1  on  a  scale 
of  1  to  5,  with  1  meaning  "not  satisfied"  and  5  meaning  "extremely  satisfied." 
Figure  2-1  presents  the  percentage  distribution  of  survey  responses  concerning 
satisfaction  with  the  quality  of  the  research.1  The  responses  on  the  quality  of  re¬ 
search  were  universal.  That  is,  the  differences  in  satisfaction  levels  reported  by 
peer  review  panelists  in  different  disciplines  (eg.,  astrophysics,  space  physics,  or 
solar  system  exploration),  with  different  amounts  of  work  experience  (eg., 
2  years  versus  10  years),  or  different  levels  of  participation  on  NASA  review  pan¬ 
els  (eg.,  2  to  4  times  versus  8  to  10  times)  are  not  statistically  significant.2 

Peer  review  panelists  also  are  fairly  satisfied  with  the  overall  science  grant 
process;  the  scores  averaged  3.2  on  the  l-to-5  satisfaction  scale.  Moreover,  the 
differences  among  panelists  from  different  disciplines  or  with  different  experi¬ 
ence  levels  were  not  statistically  significant. 

In  contrast,  proposal  writers  are  somewhat  less  satisfied  with  the  overall  sci¬ 
ence  grant  process.  Their  scores  averaged  2.7,  between  slightly  satisfied  and 
fairly  satisfied. 


1  The  3.1  score  on  the  quality  of  research  was  derived  as  a  weighted  average  of  each 
of  the  different  levels  of  satisfaction  with  the  percentages  of  respondents  for  each  satis¬ 
faction  level  serving  as  the  weights.  The  calculation  is  as  follows:  3.05  =  1  x  0.0881  +  2  x 
0.1503  +  3  x  0.4093  +  4  x  0.3264  +  5  x  0.0269. 

2  We  used  regression  analysis  to  formally  test  whether  satisfaction  with  the  overall 
grant  process  varies  consistently  with  the  background  of  the  peer  review  panelists.  Each 
peer  review  panelist's  satisfaction  score  on  the  overall  grant  process  was  regressed 
against  categorical  "dummy"  factors  to  take  account  of  the  discipline  of  the  peer  review 
panelist,  the  panelist's  years  of  experience  in  those  disciplines,  and  the  reviewer's  years 
of  participation  on  peer  panels.  None  of  these  background  factors  was  statistically  sig¬ 
nificant  at  the  upper  1  percent  of  the  F  distribution. 
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LEGEND 


1  =  Not  Satisfied 

2  =  Slightly  Satisfied 

3  =  Fairly  Satisfied 

4  =  Very  Satisfied 

5  =  Extremely  Satisfied 


Figure  2-1. 

Level  of  Satisfaction  That  Grant  Process  Results  in  Best  Research 


Figure  2-2  compares  the  levels  of  satisfaction  with  the  overall  science  grant 
process  reported  by  the  peer  review  panelists  with  those  reported  by  the  pro¬ 
posal  writers.  On  average,  the  satisfaction  level  of  proposal  writers  was  half  a 
grade  lower  than  that  of  peer  review  panelists  —  2.7  versus  3.2  on  the  l-to-5  sat¬ 
isfaction  scale.  However,  the  satisfaction  level  of  proposal  writers  who  were 
relatively  successful  in  obtaining  grants  was  about  the  same  as  that  of  tine  peer 
review  panelists;  successful  writers  scored  an  average  of  3.0,  indicating  that  they 
are  fairly  satisfied.  In  contrast,  proposal  writers  who  were  less  successful  than 
the  average  in  obtaining  grants  registered  a  satisfaction  level  of  only  2.4.  The  dif¬ 
ference  based  on  success  in  receiving  grant  awards  is  statistically  significant. 
(The  average  survey  respondent  had  a  40  percent  success  rate  in  obtaining 
awards.) 

Satisfaction  levels  with  the  overall  grant  process  are  explained  by  the  quanti¬ 
tative  answers  to  more  detailed  survey  questions  and  by  almost  70  pages  of  com¬ 
ments  and  suggestions  provided  by  the  survey  respondents.  Both  the  proposal 
writers  and  the  peer  review  panelists  suggested  that  the  current  science  grant 
process  worked  better  when  grant  funding  was  considerably  higher  than  it  is 
now.  They  also  suggested  that  the  current  science  grant  process  has  problems 
and  needs  to  be  changed  accordingly. 

The  behavior  of  proposal  writers  and  peer  review  panelists  helps  explain 
why  they  want  changes  to  the  current  science  grant  process.  In  general,  the 
number  of  proposals  being  written  and  requiring  review  has  increased  as  science 
grant  funding  has  dropped.  Many  researchers  stated  that  they  have  been  writing 
more  proposals  to  improve  their  chances  of  receiving  a  grant  award.  They  be¬ 
lieve  that  they  must  write  five  or  more  proposals  to  ensure  the  receipt  of  a  single 
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Comparison  of  Levels  of  Satisfaction  with  the  Overall  Grant  Process 


grant  award  from  NASA.  Moreover,  they  commented  that  the  amount  of  time 
they  must  spend  writing  proposals  consumes  substantial  amounts  of  their  pro¬ 
ductive  research  time. 

Likewise,  peer  review  panelists  indicated  that  they  are  overloaded  with  pro¬ 
posals  for  review  both  as  lead/ secondary  panelists  and  as  regular  panel  mem¬ 
bers.  The  survey  shows  that,  on  average,  each  lead/secondary  reviewer  is 
required  to  critique  seven  proposals  at  a  panel  meeting.  Many  lead/ secondary 
panel  reviewers  believe  that  their  effectiveness  is  hampered  when  faced  with 
more  than  five  critiques  at  a  panel  meeting. 

NASA  is  also  feeling  the  strains  of  the  large  numbers  of  proposals  that  must 
be  processed.  It  now  takes  an  average  of  more  than  13  months  from  the  time 
NASA  issues  an  NRA  to  the  time  it  disburses  the  funds  to  the  grantee:  3  months 
for  preparation  time,  6.5  months  for  evaluation  feedback  time  and  notification, 
and  4  or  more  months  to  receive  the  funding  from  NASA  and  the  grantee's  uni¬ 
versity.  Earlier,  when  grant  funding  levels  were  considerably  higher,  processing 
times  were  considerably  lower  and  more  manageable. 

Qualitatively,  both  the  peer  review  panelists  and  proposal  writers  believe 
that  the  large  numbers  of  proposals  being  written  and  reviewed  have  jeopard¬ 
ized  the  fairness  of  the  current  science  grant  process.  Peer  review  panelists  indi¬ 
cated  that  they  are  fairly  satisfied  with  the  fairness  of  the  peer  panel  review 


2-3 


process;  their  scores  averaged  3.2  on  the  l-to-5  satisfaction  scale.  At  the  same 
time,  because  of  the  increases  in  the  volume  of  proposal  reviews,  many  peer  re¬ 
view  panelists  indicated  that  they  no  longer  have  the  time  to  fully  read  and  di¬ 
gest  all  of  the  proposals  before  panel  meetings.  They  suggested  that  the 
lead/secondary  reviewer  often  has  considerably  more  weight  than  is  desirable  in 
recommending  grant  awards,  especially  when  the  panel  has  not  relied  on  mail-in 
reviews  to  obtain  additional  independent  views. 

Also,  many  peer  review  panelists  believe  that  the  evaluation  methods  need 
to  be  refined  to  enable  them  to  rate  the  quality  of  the  proposals  more  easily. 
They  indicated  that  the  current  peer  review  process  for  categorizing  proposals  is 
suitable  for  identifying  the  very  good  from  the  very  bad  proposals  but  not  very 
suitable  for  shedding  light  on  the  majority  of  proposals  that  are  neither  very 
good  nor  very  bad  in  quality.  As  a  result,  peer  review  panelists  said  that  panels 
spend  a  great  deal  of  their  time  justifying  why  they  do  not  recommend  certain 
proposals  for  funding. 

Proposal  writers  were  even  less  satisfied  with  the  fairness  of  the  science 
grant  process;  their  scores  averaged  2.7  on  the  l-to-5  satisfaction  scale.  It  is  un¬ 
derstandable  that  proposal  writers  who  are  not  frequent  recipients  of  grants 
might  think  the  process  was  not  fair.  However,  many  proposal  writers  (even 
those  who  have  been  very  successful  in  receiving  NASA  grants)  suggested  that 
the  science  grant  process  favors  proposals  that  come  from  well-known  principal 
investigators  at  the  larger,  more  distinguished  universities.  Proposal  writers  also 
mentioned  other  concerns,  such  as  the  inability  of  overworked  panelists  to  do  a 
full  and  complete  review;  lack  of  emphasis  on  using  mail-in  reviews  to  obtain 
additional  independent  views;  and  incomplete  feedback  on  scientific  merit, 
budget,  mission,  and  other  award  considerations. 


Improvement  Targets 

The  survey  results  indicate  that  NASA  should  adopt  new  procedures,  proc¬ 
esses,  and  technologies  for  reducing  unnecessary  proposal  writing,  for  lowering 
review  burdens,  and  for  enhancing  the  fairness  of  the  science  grant  process. 
Some  of  these  changes  can  be  made  quickly,  but  others  may  take  a  year  or  more 
to  complete. 

One  change  that  NASA  can  make  quickly  is  to  update  its  time  standards  for 
processing  grants.  Currently,  the  science  grant  process  takes  about  13  months. 
The  proposal  writers  have  suggested  that  the  grant  processing  time  be  reduced 
to  no  more  than  9  months:  3  months  for  proposal  preparation;  4  months  for 
evaluation  feedback  and  notification;  and  no  more  than  2  months  to  disburse  the 
funding  to  the  grantee.  These  standards  should  be  reported  to  the  National  Per¬ 
formance  Review  and  published  in  each  NRA. 

It  is  one  thing  to  develop  stricter  time  standards  and  disseminate  them,  it  is 
another  thing  to  meet  those  tougher  requirements  consistent  with  the  National 
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Performance  Review  and  NASA's  strategy  of  "faster,  better,  and  cheaper."3  That 
strategy  means  that  the  science  grant  process  will  have  to  propose,  review,  and 
fund  grants  more  quickly;  to  judge  proposals  more  fairly;  and  to  process  them 
more  efficiently.  Survey  respondents  provided  NASA  with  many  sensible  sug¬ 
gestions  for  improving  the  science  grant  process  and  meeting  the  objective  of 
"faster,  better,  and  cheaper." 

To  reduce  workload,  NASA  could  begin  by  reviewing  the  program  scope  in 
its  research  announcements.  The  program  scope  in  the  NRAs  now  provides  only 
broad  statements  about  what  is  included  in  the  overall  research  program  of  the 
announced  area.  It  apparently  does  not  provide  enough  guidance  to  would-be 
proposal  writers  about  NASA's  research  priorities  and  the  share  of  funding  as¬ 
signed  to  those  areas.  As  a  result,  proposal  writers  seem  to  be  preparing  and 
submitting  proposals  in  areas  that  are  not  justified  by  NASA's  funding  levels  in 
those  areas,  and  peer  review  panelists  seem  to  be  allocating  valuable  time  for  re¬ 
viewing  some  proposals  that  are  unlikely  to  receive  funding.  As  is  done  for 
many  other  Federal  grant  programs,  NASA  should  be  specific  when  stating  the 
scope  of  a  research  program;  the  NRA  scope  should  include  the  research  priori¬ 
ties  in  the  announced  area  and  specify  the  maximum  funding  levels  for  each  of 
those  priority  areas.  If  NASA  were  to  include  a  more  fully  developed  program 
scope  in  its  research  announcements,  proposal  writers  could  be  more  selective 
when  deciding  whether  to  respond  to  a  particular  NRA,  and  with  fewer  propos¬ 
als  to  review,  peer  review  panelists  could  be  more  effective.  Moreover,  the  re¬ 
duction  in  unnecessary  proposal  writing  and  peer  panel  reviewing  would  not 
affect  the  quality  of  NASA's  grant  research  program. 

NASA  can  also  reduce  its  review  workloads,  lower  costs,  and  enhance  fair¬ 
ness  by  modifying  its  peer  review  process  slightly.  The  current  science  grant 
process  essentially  subjects  all  proposals  to  peer  review  panels,  but  that  process 
has  resulted  in  too  many  proposals  for  an  effective  panel  review.  NASA  can  im¬ 
prove  the  peer  review  process  by  instituting  a  two-step  science  grant  process: 

♦  Subject  all  proposals  to  three  or  more  independent  and  anonymous  mail-in 
reviews  to  identify  the  proposals  that  are  technically  not  acceptable  for 
funding.  Management  cost  plans  would  not  be  needed  to  accomplish  this 
initial  technical  review. 

♦  Submit  the  remaining  proposals  that  are  acceptable  for  funding  (with  man¬ 
agement  cost  plans)  to  peer  panelists  for  further  review  and  recommenda¬ 
tion.  The  remaining  parts  of  the  process  would  remain  the  same. 

The  two-step  review  process  has  a  number  of  advantages.  It  could  reduce 
the  workloads  of  peer  review  panelists,  perhaps  by  30  percent,  thereby  allowing 
them  to  spend  the  necessary  time  to  more  effectively  review  and  recommend  de¬ 
serving  proposals  for  funding.  Also,  it  could  broaden  the  participation  of  others 
in  the  peer  review  process  and  enhance  the  reality  and  perception  of  fairness. 


3  NASA,  Federal  Laboratory  Review,  February  1995. 
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NASA  can  also  reduce  the  burdens  on  proposal  writers  and  peer  review 
panelists  by  using  information  technologies  in  the  science  grant  process.  The 
survey  respondents  strongly  supported  the  idea  of  using  information  technolo¬ 
gies  to  prepare  proposals  (80  to  90  percent  gave  their  approval),  submit  propos¬ 
als  (66  to  76  percent  gave  their  approval),  and  notify  proposal  writers  of  the 
results  (70  to  80  percent  gave  their  approval).  The  peer  review  panelists  also  in¬ 
dicated  that  they  needed  more  personal  computers  at  panel  meetings.  NASA 
should  review,  study,  and  apply  automation,  Internet,  and  electronic  data  inter¬ 
change  to  its  science  grant  process,  as  appropriate.  (Other  agencies  have  recently 
begun  efforts  to  introduce  information  technologies  to  their  grant  processes,  but 
no  consensus  on  the  most  desirable  approach  has  been  reached  as  yet.) 

NASA  should  also  consider  three  other  steps  for  improving  the  science  grant 
process.  First,  a  number  of  survey  respondents  indicated  that  fairness  would  be 
improved  substantially  if  proposals  were  reviewed  without  the  identification  of 
the  investigators  and  their  institutions.  Such  blind  reviews  would  help  dispel 
the  belief  that  established  investigators  and  institutions  are  favored  by  the  cur¬ 
rent  process. 

Second,  to  improve  fairness  and  reduce  workload,  a  number  of  survey  re¬ 
spondents  believe  that  each  proposal  should  be  scored  and  ranked  by  the  peer 
panel  reviews.  To  accomplish  this,  each  evaluation  criterion  could  be  scored 
from  0  to  10  and  summed  to  obtain  an  overall  score  for  each  proposal,  and  then 
proposals  could  be  ranked  by  numerical  scores.  A  formal  scoring  methodology 
would  enable  the  peer  review  panelists  to  rank  the  various  proposals.  Automa¬ 
tion  would  undoubtedly  be  helpful  in  implementing  such  a  scoring  methodol¬ 
ogy- 


Third,  NASA  should  consider  providing  proposal  writers  with  more  com¬ 
plete  evaluation  feedback.  Other  agencies  provide  detailed  information  about 
specific  proposal  weaknesses  or  deficiencies  as  part  of  their  evaluation  feedback, 
and  some  offer  a  face-to-face  debriefing  upon  written  request.  Not  only  do  pro¬ 
posal  writers  want  detailed  reviews,  they  also  want  to  be  informed  of  all  factors 
that  are  involved  in  award  decisions,  including  scientific  merit,  programmatic 
considerations,  budget  limitations,  and  other  relevant  factors.  Such  additional 
information  could  help  to  enhance  the  perception  of  fairness  as  well  as  to  im¬ 
prove  future  proposals  submitted  to  NASA. 
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Chapter  3 


Survey  Conduct 


To  conduct  the  surveys  of  the  proposal  writers  and  peer  review  panelists, 
LMI  selected  the  mail  as  the  most  appropriate  medium.  Like  any  type  of  survey, 
mail  surveys  have  both  advantages  and  disadvantages.1  However,  in  the  case  of 
the  science  grant  surveys,  we  considered  the  mail  to  be  the  only  practical,  and 
the  least  costly,  medium,  because  the  would-be  respondents  reside  throughout 
the  United  States  and  in  foreign  countries.  Another  advantage  of  a  mail  survey 
is  that  it  can  be  accompanied  by  a  cover  letter,  which  can  enhance  credibility  and 
trust. 

Conducting  a  survey  by  mail  requires  careful  design  of  the  questionnaire  to 
ensure  that  the  correct  questions  are  asked  and  that  the  wording  is  clear.  In  ad¬ 
dition,  this  type  of  survey  requires  follow-up  post  cards  and  letters  to  ensure  a 
high  response  rate.  This  chapter  discusses  the  design  of  surveys,  as  well  as  selec¬ 
tion  of  the  samples,  administration  of  the  surveys,  and  response  rates. 


Survey  Design 

Separate  questionnaires  were  needed  for  proposal  writers  and  peer  review 
panelists  in  the  NASA  science  grant  process.  Although  proposal  writers  and 
peer  review  panelists  are  both  scientists  and  direct  external  customers  of  the 
NASA  science  grant  process,  their  perspectives  differ  greatly.  For  example,  pro¬ 
posal  writers  are  concerned  about  the  time  they  have  for  preparing  the  proposals 
and  for  obtaining  feedback,  while  peer  review  panelists  are  concerned  about  the 
time  they  have  for  evaluating  the  proposals  and  for  presenting  their  findings. 
Also,  the  guidelines  for  preparing  and  submitting  the  proposals  are  entirely  dif¬ 
ferent  from  the  guidelines  for  evaluating  and  rating  the  proposals.  Because  of 
these  wide  differences,  we  developed  separate  questionnaires  for  the  peer  review 
panelists  and  proposal  writers. 

We  used  a  variety  of  inputs  for  designing  these  science  grant  surveys.  First, 
we  held  numerous  discussions  with,  and  received  much  documentation  from, 
the  Office  of  Space  Science.  Second,  we  independently  observed  the  NASA  space 
science  peer  review  process  to  obtain  first-hand  knowledge  of  how  it  operates. 
Third,  during  three  separate  survey  pretests,  we  had  discussions  with  peer  re¬ 
view  panelists  and  proposal  writers  about  their  grant  process  concerns.  Fourth, 
we  followed  various  design  principles  in  developing  the  survey.  These  survey 
design  principles  included  presenting  the  survey  in  a  7"x  SVz"  booklet  to  avoid 

1See,  for  example,  Louis  M.  Rea  and  Richard  A.  Parker,  Designing  and  Conducting 
Survey  Research:  A  Comprehensive  Guide,  San  Francisco:  Jossey-Bass  Publishers,  1992.  It 
discusses  the  use  of  the  mail,  telephone,  and  in-person  interviews  to  conduct  a  survey. 
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giving  the  impression  that  the  survey  is  difficult  and  time-consuming  to  com¬ 
plete;  reserving  the  front  and  back  booklet  covers  to  state  the  purpose  of  the  sur¬ 
vey  and  to  elicit  interest;  constructing  close-ended,  multiple  choice  questions, 
rather  than  open-ended  questions,  to  minimize  the  time  for  survey  completion; 
grouping  the  questions  under  topic  headings  to  maintain  respondent  interest; 
and  putting  the  easy  and  more  interesting  questions  ahead  of  the  more  difficult 
and  less  interesting  questions,  also  to  maintain  interest. 


Questionnaires 

The  proposal  writer  survey  contains  15  basic  questions  that  seek  to  measure 
satisfaction  levels,  customer  service  standards,  and  the  relative  importance  that 
customers  attach  to  grant-process  quality  factors.  Appendix  D  contains  the  pro¬ 
posal  writer  survey.  Some  questions  focus  on  the  time  for  preparing  grant  pro¬ 
posals,  obtaining  proposal  feedback,  and  executing  research,  and  others  focus  on 
NASA's  guidelines  for  responding  to  NRAs  and  other  guidance.  In  addition,  the 
survey  asks  some  general  questions  relating,  for  example,  to  the  relative  impor¬ 
tance  that  the  proposal  writers  attach  to  feedback,  fairness,  and  processing  time, 
as  well  as  to  the  background  of  the  respondents.  Finally,  a  comment  section  asks 
for  suggestions  that  the  proposal  writers  might  have  for  improving  NASA's  sci¬ 
ence  grant  process. 

The  questionnaire  for  peer  review  panelists  contains  18  basic  questions  and, 
like  the  proposal  writer  survey,  seeks  to  measure  customer  satisfaction  levels, 
customer  service  standards,  and  the  relative  importance  that  customers  attach  to 
grant-process  quality  factors.  Appendix  E  displays  the  peer  review  panelist  sur¬ 
vey.  Its  questions  focus  on  the  time  and  workload  for  reviewing  grant  proposals; 
on  NASA's  guidelines  for  evaluating  and  rating  proposals;  and  on  NASA's  sup¬ 
port  of  the  peer  review  process,  including  expense  reimbursement  and  use  of  in¬ 
formation  technology.  Other,  more  general  questions  concern  the  qualifications 
of  fellow  panel  members,  procedures  for  avoiding  conflicts  of  interest,  and  im¬ 
portance  that  peer  review  panelists  attach  to  various  quality  factors.  The  survey 
also  asks  about  the  background  of  the  peer  review  panelists  and  has  a  comment 
section  for  the  respondents  to  make  suggestions  for  improving  NASA's  science 
grant  process. 


Pretests 


We  held  three  separate  survey  pretests  of  the  survey  instruments,  one  in 
Houston,  Texas,  and  two  in  Washington,  D.C.  As  a  result  of  discussions  with  the 
peer  review  panelists  and  proposal  wxiters  who  participated  in  the  pretests,  we 
changed  the  wording  of  a  few  questions,  dropped  one  question  entirely,  and 
added  a  couple  of  questions  to  the  survey  instruments.  Also,  we  verified  that  the 
respondents  could  answer  the  questionnaires  within  10  minutes,  well  within  the 
15-minute  limitation  established  by  OMB. 
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In  addition  to  testing  the  questionnaires  in  the  pretests,  we  verified  that  the 
response  rates  would  be  relatively  low,  under  30  percent,  if  we  contacted  respon¬ 
dents  only  once.  Hence,  we  concluded  that  we  would  need  to  contact  the  would- 
be  respondents  several  times  to  obtain  satisfactory  response  rates. 


Post  Cards  and  Letters 

To  explain  the  purpose  of  the  surveys  and  to  elicit  cooperation  from  would- 
be  respondents,  we  developed  two  post  cards  and  two  letters  that  could  be  sent 
to  both  the  proposal  writers  and  the  peer  review  panelists.  Using  generic  post 
cards  and  letters  helped  minimize  the  survey  costs.  Appendix  F  contains  these 
four  communications: 

1.  Post  card  used  to  alert  the  selected  participants  that  the  survey  would  be 
coming  in  about  a  week,  that  cooperation  is  needed  for  completing  the  sur¬ 
vey,  and  that  each  opinion  matters. 

2.  Cover  letter  that  accompanied  the  surveys  to  explain  the  importance  of  the 
effort,  to  underscore  the  need  for  participation  for  obtaining  a  representative 
sample  of  views,  to  guarantee  confidentiality,  and  to  offer  a  summary  of 
survey  results  to  each  participant. 

3.  Post  card  sent  one  week  later  to  thank  those  who  had  already  responded  as 
well  as  to  remind  those  who  had  not  yet  responded  to  do  so  and  to  offer 
them  another  copy  of  the  questionnaire  that  they  could  obtain  by  placing  a 
telephone  call. 

4.  Final  letter,  and  another  copy  of  the  questionnaire,  sent  only  to  nonrespon¬ 
dents  about  two  weeks  later.  The  purpose  of  the  letter  was  to  inform  nonre¬ 
spondents  that  that  their  questionnaires  had  not  been  received  and  to 
reiterate  the  importance  of  the  survey,  stress  the  urgency  of  receiving  all 
completed  surveys  to  ensure  that  the  community's  opinions  are  accurately 
represented,  and  the  offer  of  a  summary  of  survey  results. 

The  close  timing  between  the  post  card  and  letter  mailings  was  chosen  because 
survey  research  has  found  that  survey  participants  usually  respond  within  a  few 
days  of  receipt  of  a  questionnaire  or  a  letter  concerning  the  questionnaire. 
Hence,  waiting  longer  periods  between  mailings  does  not  improve  response 
rates  but  merely  wastes  time. 


Sample  Selection 

The  Office  of  Space  Science  provided  FY93  mailing  lists  of  the  peer  review 
panelists  and  proposal  writers  for  selecting  samples.  LMI  drew  samples  from 
these  mailing  lists  to  bound  sampling  error  within  plus  or  minus  3  percentage 
points  at  the  95  percent  confidence  limit.  We  stratified  the  sample  only  when 
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there  was  reason  to  believe  that  the  survey  questions  might  be  answered  differ¬ 
ently  by  different  strata  in  the  population.  We  used  systematic  random  sampling 
(selecting  every  nth  one)  of  alphabetized  name  listings  to  select  the  samples.2 

Selection  of  peer  review  panelists  to  participate  in  the  survey  was  straight¬ 
forward.  The  FY93  mailing  list  showed  a  population  of  540  peer  review  panel¬ 
ists.  Working  with  the  hypergeometric  distribution,  as  required  to  draw  a 
sample  from  such  a  small  population,  we  took  a  50  percent  systematic  random 
sample.  The  resulting  sample  of  270  peer  review  panelists  would  provide  a  mar¬ 
gin  of  error  of  plus  or  minus  3  percentage  points  at  the  95  percent  confidence 
limit.3  Stratified  sampling  was  not  necessary  for  peer  review  panelists. 

The  selection  process  was  somewhat  more  difficult  for  the  proposal  writers. 
NASA's  FY93  mailing  list  showed  a  population  of  1,316  proposal  writers,  but 
NASA  had  reason  to  believe  that  proposal  writers  who  had  been  consistently  de¬ 
nied  grant  awards  during  FY93  might  provide  different  answers  to  some  ques¬ 
tions  than  those  who  had  received  one  or  more  grant  awards  during  that  period. 
Therefore,  we  stratified  the  proposal  writer  sample  accordingly.  Table  3-1  sum¬ 
marizes  the  population  and  stratified  sampling  information  for  the  proposal 
writers. 

Table  3-1. 

Selection  of  Proposal  Writer  Sample 


Population 

Sample 

FY93  award  experience 

Number 

Percentage 

Number 

Percentage 

Grants  awarded 

562 

42.7 

42.7 

No  grants  awarded 

754 

57.3 

57.3 

Total 

1,316 

100.0 

658 

100.0 

As  shown  in  Table  3-1,  we  divided  the  1,316  proposal  writers  into  two 
groups:  562  who  had  been  awarded  at  least  one  grant  and  754  who  had  not  been 
awarded  any  grants  during  FY93.  With  the  same  statistical  concepts  and  sam¬ 
pling  procedures  used  for  selecting  the  peer  review  panelists  to  participate  in  the 
survey,  we  drew  281  proposal  writers  who  had  been  awarded  one  or  more 
grants  (50  percent  of  562)  and  377  who  had  not  been  awarded  a  grant  (50  percent 
of  754).  As  a  result,  the  658  proposal  writers  in  the  sample  are  representative  of 
the  proposal  writers  in  the  population  who  had  received  one  or  more  grants 
(42.7  percent  of  both  the  population  and  the  sample)  and  of  the  proposal  writers 


2  If  a  50  percent  sample  was  needed,  we  drew  every  other  name  on  the  alphabetized 
list.  However,  in  this  case,  we  first  randomly  picked  whether  to  start  the  drawing  of  the 
sample  with  the  first  or  second  name  on  the  mailing  list. 

3  If  we  achieve  a  75  percent  response  rate,  the  sampling  error  would  be  bounded  be¬ 
tween  plus  or  minus  5  percentage  points,  still  an  acceptable  margin  of  sampling  error  for 
this  survey  effort. 
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who  had  not  been  awarded  any  grants  in  FY93  (57.3  percent  of  both  the  popula¬ 
tion  and  the  sample). 

In  total,  we  selected  928  individuals  to  participate  in  the  NASA  science  grant 
surveys:  270  peer  review  panelists  and  658  proposal  writers.  Of  the  928  indi¬ 
viduals  in  the  samples,  34,  or  3.7  percent,  were  selected  to  receive  both  the  pro¬ 
posal  writer  and  the  peer  review  panelist  surveys. 


Survey  Administration 

As  required  by  Federal  printing  regulations,  NASA  produced  the  7"x  8V2" 
survey  booklets  as  well  as  the  post  cards  and  letters.  LMI  provided  camera- 
ready  versions  of  the  questionnaires  and  electronic  versions  (ASCII  and  Word¬ 
Perfect  6.0  for  Windows)  of  the  draft  texts  for  the  post  cards  and  letters.  NASA 
revised  the  post  card  and  letter  texts  slightly  and  added  the  NASA  logo  to  those 
communications .  NASA  also  mailed  all  of  the  questionnaires,  post  cards,  and 
letters,  and  it  established  a  special  mailbox  for  receipt  of  survey  returns. 

So  that  survey  responses  could  be  tracked,  NASA  assigned  a  three-digit 
code  to  each  individual  selected  to  participate  in  the  surveys.  That  code  was 
placed  on  the  survey  instrument,  the  cover  letter,  and  the  mailing  labels.  The 
codes  were  used  only  for  tracking  responses;  they  were  not  used  for  identifying 
the  respondents  with  particular  answers  to  questions.  NASA  prepared  each 
would-be  survey  respondent's  coded  survey  instrument  in  duplicate  for  a  possi¬ 
ble  follow-up. 

NASA  mailed  all  survey  instruments  and  communications  on  Wednesdays 
because  it  is  a  relatively  light  mailing  day.  It  mailed  the  alert  post  card  on 
Jan.  25,  1995.  On  Feb.  1, 1995,  it  mailed  the  survey  instrument,  the  cover  letter, 
and  a  self-addressed  meter-postage  return  envelope  —  all  contained  in  a  stan¬ 
dard  Number  10  envelope.  NASA  mailed  the  thank-you/reminder  post  cards  to 
the  928  individuals  in  the  samples  on  Feb.  8,  1995.  On  Feb.  24,  1995,  NASA 
mailed  the  final  appeal  to  any  individuals  who  had  not  responded.  However, 
those  who  had  telephoned  for  a  second  questionnaire  in  response  to  the  re¬ 
minder  card  were  not  sent  a  final  appeal  letter  because  they  had  demonstrated 
their  interest  in  the  survey  and  would  most  likely  complete  the  questionnaire. 

After  the  final  appeal  letter  was  mailed,  we  waited  four  weeks  —  until 
Mar.  22, 1995  —  before  "closing  the  books"  on  the  period  for  receiving  the  data. 

The  survey  data  were  entered  into  separate  DBase  (.dbf)  files  on  PC  disk¬ 
ettes.  To  facilitate  data  entry,  we  assigned  numerical  codes  for  each  category  of 
response  on  the  questionnaire.  (Those  codes  did  not  appear  on  the  survey  in¬ 
struments  that  were  sent  to  the  respondents.)  Our  data  entry  staff  verified  the 
accuracy  of  the  data  entries  by  repunching  25  percent  of  the  surveys  and  check¬ 
ing  for  errors.  If  an  error  was  found  anywhere,  they  repunched  another  25  per¬ 
cent  of  the  surveys  and  checked  for  errors  in  this  new  batch.  The  process 
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continued  until  an  error-free  batch  was  found  or  until  all  surveys  were  com¬ 
pletely  verified. 

For  analyzing  the  survey  data,  we  chose  the  Statistical  Analysis  System 
(SAS).  SAS  creates  data  sets  from  the  DBase  files  and  is  very  good  for  providing 
tabulations  and  graphics  of  results,  for  determining  the  statistical  relationships 
among  various  question  answers,  and  for  evaluating  whether  answers  to  ques¬ 
tions  are  universal  or  depend  upon  the  background  of  the  survey  respondents. 


Response  Rates 

We  achieved  a  73  percent  response  rate  overall  on  the  science  grant  surveys 
through  Mar.  22,  1995.4  For  the  peer  review  panelists,  we  attained  a  76  percent 
survey  response  rate  (192  questionnaires  completed  out  of  252  mailed  and  re¬ 
ceived  by  the  survey  participants).  We  recorded  a  slightly  lower  survey  re¬ 
sponse  rate  on  the  two  proposal  writer  survey  strata:  74  percent  for  those  who 
were  awarded  grants  in  FY93  (203  completed  out  of  273  mailed  and  received  by 
the  respondents)  and  70  percent  for  those  who  had  not  been  awarded  a  grant  in 
FY93  (250  completed  out  of  358  mailed  and  received  by  the  respondents). 
Table  3-2  provides  details  about  the  mailings  and  response  rates  achieved  in  the 
science  grant  surveys. 

Table  3-2. 

Mailings  and  Response  Rates  for  Science  Grant  Surveys 


Surveys  mailed  to  respondents 

Survey  strata 

Targeted 

Not 

received 

Received 

Surveys 

completed 

Response 
rate  (%)a 

Peer  review 

270 

18 

252 

192 

76 

Proposal  writer  with  FY93 
grant 

281 

8 

273 

203 

74 

Proposal  writer  without  FY93 
grant 

377 

19 

358 

250 

70 

Total 

928 

45 

883 

645 

73 

aThe  response  rate  is  calculated  as  the  number  of  surveys  completed  relative  to  the  number  of  surveys 
mailed  and  received  by  the  survey  participants. 


The  response  rates  have  implications  for  sampling  error  and  nonresponse 
bias.  A  73  percent  response  rate  is  considered  very  good  by  statisticians  for  lim¬ 
iting  bias  in  survey  answers.  A  50  to  60  percent  response  rate  is  often  considered 


4  After  our  Mar.  22, 1995,  closing  date  for  recording  survey  returns,  we  received  some 
20  additional  returns,  mostly  for  proposal  writers,  and  that  raised  the  overall  response 
rate  to  75  percent.  Although  we  did  not  formally  include  these  additional  returns  in  our 
data  base,  we  did  analyze  their  inputs. 
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satisfactory  for  the  purposes  of  analysis  and  reporting  of  findings.  In  terms  of 
sampling  error,  the  margin  of  error  is  plus  or  minus  5  percentage  points  at  the 
95  percent  confidence  limit  for  the  number  of  surveys  completed  in  each  strata. 
Thus,  if  75  percent  of  the  peer  review  survey  respondents  answered  "yes"  to  a 
survey  question,  we  can  be  sure  that  70  to  80  percent  of  all  of  the  peer  review 
panelists  in  the  population  would  answer  that  way  (95  percent  of  the  time). 

The  multiple  contacts  with  the  survey  respondents  played  an  important  part 
in  achieving  these  response  rates.  A  regression  analysis  of  21  response  rates  re¬ 
flecting  three  different  types  of  survey  respondents  and  seven  weekly  returns  — 
i.e.,  peer  review  panelists,  proposal  writers  with  FY93  grants,  and  proposal  writ¬ 
ers  without  FY93  grants,  each  tracked  over  seven  weeks  —  revealed  a  "spiked" 
time  pattern  for  the  responses.  The  spiked  time  pattern  explains  95  percent  of 
the  variation  in  response  rates,  with  no  statistically  significant  differences  shown 
between  the  types  of  respondents.  Figure  3-1  displays  the  estimated  overall  time 
pattern  of  response  rates. 
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Figure  3-1. 

Effect  of  Respondent  Contacts  on  Response  Rates 


The  spiked  time  pattern  of  response  rates  occurred  twice  during  the  conduct 
of  the  surveys.  First,  between  Jan.  25  and  Feb.  8,  we  mailed  an  alert  post  card, 
the  survey  instrument,  and  a  thank-you/ reminder  post  card  to  the  respondents. 
These  three  contacts  were  responsible  for  the  first  spiked  time  pattern:  a  4  per¬ 
cent  response  rate  on  Feb.  8,  peaking  at  30  percent  on  Feb.  15,  and  declining  to 
10  percent  on  Feb.  22  and  to  8  percent  on  Mar.  1  —  a  cumulative  52  percentage 
point  response  rate.  Second,  after  we  mailed  a  final  appeal  letter  and  second  sur¬ 
vey  instrument  on  Feb.  22,  the  response  rate  rose  to  14  percent  on  Mar.  8  then 
declined  to  6  percent  on  Mar.  15  and  1  percent  on  Mar.  22  —  an  additional 
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21  percentage  point  response  rate.  Although  the  earlier  contacts  may  have  had 
some  lingering  effect  on  the  additional  response  achieved  in  the  second  time 
spike,  it  seems  likely  that  most  of  that  increase  was  due  to  the  final  appeal  con¬ 
tact. 
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Chapter  4 


Key  Survey  Results:  Peer 
Review  Panelists 


In  this  chapter,  we  discuss  the  key  results  from  our  survey  of  peer  review 
panelists.  We  summarize  the  overall  quality  of  the  science  grant  process  from 
the  perspective  of  peer  review  panelists,  present  their  responses  to  key  survey 
questions,  and  discuss  the  implications  of  those  responses  for  improving  the  sci¬ 
ence  grant  process.  Appendix  G  shows  the  distribution  of  responses  for  each 
survey  question,  and  Appendix  H  provides  the  comments  made  by  the  respon¬ 
dents. 


Quality  Index 

From  the  survey  results,  we  computed  a  quality  index  to  show  how  well  the 
quality  standards  of  peer  review  panelists  have  been  met  and  to  identify  those 
components  of  the  science  grant  process  that  need  improvement.  We  calculated 
that  the  peer  review  panelists  rated  the  overall  quality  of  the  science  grant  proc¬ 
ess  at  71  percent,  on  a  0  to  100  percent  scale.  That  is,  on  average,  they  believe 
that  71  percent  of  their  quality  standards  have  been  fully  met.  Table  4-1  shows 
the  construction  of  the  quality  index  for  peer  review  panelists. 

The  satisfaction  scores  on  the  quality  components  for  peer  review  panelists 
ranged  from  approximately  62  to  88  percent.  We  targeted  quality  components 
for  improvement  if  they  received  an  average  score  of  less  than  70  percent.  On 
the  l-to-5  satisfaction  scale,  an  average  score  of  under  3.5  would  signal  the  need 
for  improvement.  Using  the  70  percent  rule,  we  believe  that  improvements  are 
most  needed  for  mail-in  reviews,  review  fairness,  and  guidelines.  However,  this 
does  not  necessarily  rule  out  the  need  for  improvements  in  certain  aspects  of 
other  quality  components.  We  now  provide  the  highlights  for  the  quality  com¬ 
ponents  in  the  order  in  which  they  appear  in  Table  4-1. 


Table  4-1. 

Peer  Review  Quality  Index 


Quality  component 

Percentage 
of  standard13  (1) 

Weight0 

(2) 

Contribution  to  total 
(3)  =  (1)  x  (2) 

Time/workload 

0.883 

0.141 

0.124 

Panel  support 

0.745 

0.123 

0.091 

Panel  qualifications 

0.730 

0.163 

0.119 

Ethical  issuesa 

0.697 

0.138 

0.096 

Guidelines 

0.659 

0.127 

0.084 

Review  fairness 

0.647 

0.167 

0.108 

Mail-in  reviews 

0.618 

0.141 

0.087 

Total 

— 

1.000 

0.709 

a  Ethical  issues  refer  to  avoiding  conflicts  of  interest  and  protecting  proprietary  ideas. 


b  Percentage  of  standard  is  computed  by  taking  the  average  actual  peer  review  scores  for  the  answers  to 
questions  that  fall  under  each  quality  component  —  scores  that  range  from  1 ,  or  “not  satisfied,”  to  a  maximum 
possible  score  of  5,  or  “extremely  satisfied”  —  relative  to  the  maximum  5  score  —  the  standard  of  excellence. 

c  Respondents  used  1-to-5  scores  to  register  the  degree  of  importance  that  they  attach  to  each  quality  com¬ 
ponent.  We  calculated  a  weight  for  each  quality  component  by  taking  its  total  importance  scores  relative  to 
the  grand  total  of  all  quality  component  scores. 


Time  and  Workload 

Peer  review  panelists  indicated  that  the  time  allocated  to  them  for  proposal 
reviews,  both  before  and  during  panel  meetings,  was  reasonably  satisfactory. 
On  average,  they  received  about  87  percent  of  the  time  needed  for  prepanel 
study  and  90  percent  of  the  time  needed  for  panel  deliberations.  However,  many 
respondents  indicated  that  the  amount  of  time  has  been  decreasing  because  their 
review  workloads  have  been  increasing. 

Figure  4-1  shows  the  distribution  of  lead/ secondary  panelist  workloads.  On 
average,  half  of  the  lead/secondary  panelists  have  been  critiquing  7  or  more  pro¬ 
posals  for  a  panel  meeting.  About  42  percent  of  the  lead /secondary  panelists 
have  had  workloads  of  4  to  5  proposals,  while  25  percent  of  them  have  had 
workloads  of  10  or  more  proposals.  Of  all  the  disciplines,  astrophysicists  tend  to 
have  the  largest  lead /secondary  workloads.1 

Peer  review  panelists  believed  that  their  ability  to  review  proposals  effec¬ 
tively  was  hampered  by  these  high  workload  levels.  Some  suggested  that,  to 

1  We  used  regression  analysis  to  formally  test  whether  workload  levels  varied  consis¬ 
tently  with  the  background  of  the  peer  review  panelists.  Categorical  "dummy"  variables 
were  used  to  represent  the  discipline  of  the  peer  review  panelists,  the  panelists'  years  of 
experience,  and  their  participation  on  peer  panels.  Regressing  those  background  factors 
on  the  workloads,  we  found  that  the  discipline  of  the  peer  review  panelists  was  highly 
statistically  significant  for  explaining  differences  in  workloads. 
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Percent 


Figure  4-1 . 

Lead/Secondary  Panelist  Workload 


ensure  an  effective  review,  lead/secondary  panelist  workloads  should  not  ex¬ 
ceed  five  proposals.  With  more  reasonable  workload  levels,  they  felt  that  they 
could  both  do  the  lead/ secondary  critiques  well  and  be  adequately  prepared  for 
other  reviews  as  regular  panel  members. 


Panel  Support 

Figure  4-2  shows  the  average  levels  of  peer  review  panelists'  satisfaction 
with  the  various  items  that  support  their  panel  sessions.  The  scores  for  these 
items  of  support  are  clustered  into  five  groupings.  First,  peer  review  panelists 
gave  the  highest  scores  for  meeting  facilities,  an  average  of  4.1  or  slightly  more 
than  very  satisfied.  Second,  lodging,  supplies,  and  travel  each  received  a  3.9  av¬ 
erage  rating,  slightly  less  than  very  satisfied.  Third,  clerical  support,  coordina¬ 
tion  help,  and  telephone  and  fax  facilities  were  scored  with  a  3.8  average,  also 
quite  close  to  a  rating  of  very  satisfied.  Fourth,  meals  and  per  diem  received  a 
3.6  average  score,  also  close  to  a  very  good  score.  Fifth,  personal  computer  sup¬ 
port  received  the  lowest  rating  with  an  average  score  of  3.0  —  fairly  satisfied. 

On  the  basis  of  our  70  percent  rule  (below  3.5  on  the  l-to-5  satisfaction 
scale),  we  recommend  that  NASA  improve  the  personal  computer  support  it  pro¬ 
vides  at  peer  review  panel  meetings.  Specifically,  NASA  should  make  personal 
computers  more  widely  available  at  panel  meeting  deliberations.  The  survey  re¬ 
spondents  did  not  provide  details  on  the  types  of  personal  computers  desired. 
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Figure  4-2. 

Satisfaction  with  Peer  Review  Panel  Support 


Peer  review  panelists  strongly  support  the  use  of  information  technologies 
for  various  elements  of  the  science  grant  process.  As  shown  in  Table  4-2,  NASA 
received  strong  support  for  using  information  technologies  in  proposal  prepara¬ 
tion  (80  to  90  percent  approved),  notification  (70  to  80  percent  approved),  and 
submission  (66  percent  to  76  percent  approved).  Less  support  was  shown  for  ap¬ 
plying  information  technologies  to  proposal  evaluation  and  grant  awards  for  the 
science  grant  process. 

Table  4-2. 

Support  for  Using  Information  Technologies 
in  the  Science  Grant  Process 
(percentage) 


Process  element 

Yes 

No 

Preparation 

85 

15 

Notification 

75 

25 

Submission 

71 

29 

Award 

69 

31 

Evaluation 

64 

36 

Overall 

75 

25 

Note:  Margin  of  error  is  plus  or  minus  5  percentage  points. 
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We  believe  that  NASA  should  consider  the  use  of  computer  automation,  In¬ 
ternet,  and  electronic  data  interchange  as  may  be  appropriate  to  its  science  grant 
process.  Other  Federal  agencies  —  especially  the  National  Science  Foundation, 
National  Institutes  of  Health,  and  Department  of  Energy  —  have  been  exploring 
the  use  of  these  information  technologies  for  their  grant  processes  but  have  not 
reached  a  consensus  on  the  best  technological  approach  for  processing  grants. 
NASA  should  appoint  a  task  group  to  review  these  ongoing  developments  and 
to  determine  the  efficiency  and  effectiveness  of  the  different  approaches. 


Ethical  Issues 

Peer  review  panelists  did  not  believe  that  ethical  issues  —  avoiding  conflicts 
of  interest  or  protecting  proprietary  ideas  —  were  serious  problems.  On  the 
l-to-5  satisfaction  scale,  both  ethical  issues  received  a  rating  of  3.5  —  a  score 
quite  close  to  being  very  satisfied. 

In  their  comments,  peer  review  panelists  focused  on  the  issue  of  avoiding 
conflicts  of  interest.  Appendix  H  includes  a  full  set  of  those  comments,  but  we 
quote  only  three  of  them  here: 

"Concerns  about  conflict  of  interest  from  individuals  from  the  same  institution 
are  given  too  much  weight  —  interfere  with  giving  proposals  a  balanced  re¬ 
view  by  a  full  panel 

"I  felt  that  there  was  some  strong  bias  on  at  least  one  committee  members  part 
against  proposals  competing  with  those  from  that  member's  institution,  and 
that  this  greatly  affected  the  outcome  of  this  review." 

"No  panel  is  ever  free  from  all  conflicts  of  interest  —  the  fields  are  too  small." 

The  peer  review  panelists  did  not  comment  at  all  on  protecting  proprietary 
ideas. 


Guidelines 


Peer  review  panelists  were  not  very  satisfied  with  NASA  guidelines  for  re¬ 
viewing  proposals.  Figure  4-3  presents  the  average  satisfaction  levels  for  guide¬ 
lines  on  evaluation  criteria,  methods,  and  ratings.  Each  guideline  received  an 
average  rating  below  70  percent  of  the  standard  of  excellence  (under  3.5  on  the 
l-to-5  satisfaction  scale). 
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Methods 

Rating 
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Average  satisfaction  level 


Figure  4-3. 

Satisfaction  with  Evaluation  Guidelines 


Peer  review  survey  respondents  made  many  useful  comments  and  sugges¬ 
tions  for  improving  the  guidelines  for  evaluating  proposals.  Appendix  H  con¬ 
tains  the  full  set  of  those  comments;  we  quote  five  of  them  here: 

"The  evaluation  process,  particularly  guidelines,  has  improved  greatly  over  the 
years." 

"There  are  no  uniform  standards  across  programs.  There  should  be." 

"Proposal  rating  could  be  improved  by  asking  for  ratings  on  each  criterion  and 
NASA  using  them  to  arrive  at  an  overall  rating." 

"I  feel  a  fair  amount  of  time  is  actually  more  or  less  wasted  going  through  a 
formal  review  of  those  proposals  that  are  obviously  not  going  to  be  accepted.  A 
two-stage  process  would  probably  be  more  efficient.  Stage  1,  [with  independ¬ 
ent  mail-in  reviews  or  whatever]:  weed  out  the  proposals  that  will  not  be  seri¬ 
ous  competitors.  Stage  2,  convene  a  (smaller)  panel  to  deal  with  the 
[remaining]  proposals ....  [This  two-stage  procedure]  would  probably  save 
30%  of  the  actual  time  spent  on  reviews." 

We  believe  that  these  suggestions  and  others  provided  by  the  peer  review 
panelists  will  help  to  improve  the  review  process  substantially.  If  NASA  stan¬ 
dardized  its  guidelines,  peer  review  panelists  could  serve  on  different  panels 
more  easily.  If  they  had  a  scoring  methodology  for  each  criterion,  panelists 
could  more  easily  justify  their  selection  of  winning  proposals  among  the  many 
fine  proposals  that  they  receive  for  review.  Finally,  having  a  prepanel  stage  in 
the  review  process  would  substantially  reduce  the  burdens  on  peer  panelists; 
they  could  therefore  review  the  proposals  more  effectively  and  ensure  that  their 
recommendations  on  funding  are  well  justified. 


Review  Fairness 


On  average,  peer  review  panelists  were  moderately  satisfied  with  the  fair¬ 
ness  of  the  science  grant  process.  The  average  score  was  3.2,  or  just  above  fairly 
satisfied  on  the  l-to-5  satisfaction  scale.  Figure  4-4  shows  the  full  distribution  of 
responses  for  satisfaction  with  the  review  process.  It  is  understandable  numeri¬ 
cally  why  the  average  score  on  review  fairness  is  just  above  fairly  satisfied.  The 
majority  of  the  peer  review  panelists  —  45  percent  —  said  they  were  fairly  satis¬ 
fied  (and  virtually  established  the  average),  and  37  percent  said  they  were  either 
very  satisfied  or  extremely  satisfied.  Only  17  percent  of  the  peer  review  panelists 
said  they  were  either  not  satisfied  or  only  slightly  satisfied,  pulling  down  the  av¬ 
erage. 


Percent 


Level  of  satisfaction 


Figure  4-4. 

Satisfaction  with  Review  Fairness:  Peer  Review  Panelists 


The  peer  review  panelists  offered  comments  on  and  suggestions  for  improv¬ 
ing  the  fairness  of  the  review  process  (see  Appendix  H).  We  quote  six  comments 
here: 


"l  believe  [the  process]  is  fair  and  the  reviewers  [are]  largely  free  of  personal 
bias  in  their  assessments/' 

"The  last  panel  I  served  on  had  several  proposals  with  no  mail-in  reviews. 
This  puts  the  panel  in  a  position  of  having  to  review  proposals  for  which  there 
may  be  no  expertise  on  the  panel.  " 

"Make  sure  that  there  is  an  adequate  number  of  external  mail-in  reviews  (ap¬ 
proximately  four),  so  that  no  single  reviewer  carries  too  much  weight/' 
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"Too  much  favoritism  towards  established  investigators. " 

"[Use]  blind  reviews." 

"In  some  cases,  the  proposal  rating  was  only  determined  (voted)  by  the  pri¬ 
mary  and  secondary  reviewers  rather  than  the  entire  panel.  To  ensure  fairness, 
the  rating  of  every  proposal  should  be  voted  on  by  all  the  panel  members,  who 
also  [should]  read  the  proposal." 

While  most  peer  review  panelists  were  fairly  satisfied  with  the  fairness  of 
the  grant  review  process,  they  believe  that  there  is  room  for  improvement.  Many 
believe  that  independent  mail-in  reviews  should  be  mandatory  and  should  play 
a  larger  role  in  the  review  process.  Also,  many  believe  that  well-known  principal 
investigators  and  larger  institutions  are  favored;  they  suggested  that  identifica¬ 
tions  be  removed  from  the  proposals  before  they  are  given  to  reviewers.  (As  dis¬ 
cussed  in  Chapter  5,  proposal  writers  had  similar  concerns.) 

Also,  we  believe  that  the  fairness  of  the  grant  process  has  been  adversely  af¬ 
fected  by  the  large  number  of  proposals  that  must  be  reviewed.  Lead/ secondary 
reviewers  commented  that  they  have  difficulty  finding  the  time  to  critique  seven 
proposals  for  a  panel  meeting,  let  alone  finding  the  time  to  read  other  proposals 
that  they  must  assess  as  regular  members  of  other  panels.  Regular  panel  mem¬ 
bers  have  indicated  that  they  have  recently  been  overwhelmed  with  review 
workloads,  thereby  hampering  the  quality  of  their  deliberations.  Those  who 
have  served  as  independent  mail-in  reviewers  also  have  complained  that  they 
cannot  undertake  so  many  proposal  reviews  because  they  are  so  busy  with  their 
own  research  and  the  ever-growing  need  to  write  proposals.  Thus,  the  fairness 
of  the  process  will  be  helped  by  reducing  proposal  workloads. 


Mail-In  Reviews 

The  peer  review  panelists  were  only  fairly  satisfied  with  the  mail-in  review 
process.  They  gave  the  mail-in  reviews  a  numerical  score  of  3.1  on  the  l-to-5  sat¬ 
isfaction  scale.  As  indicated  in  our  discussion  of  peer  review  fairness,  many  be¬ 
lieve  that  NASA  should  require  mail-in  reviews  as  part  of  the  grant  process.  On 
the  other  hand,  many  indicated  that  the  quality  of  the  mail-in  reviews  is  uneven. 

Peer  review  panelists  thought  that  NASA  could  do  a  number  of  things  to 
improve  the  quality  of  mail-in  reviews.  First,  many  indicated  that  NASA  could 
strengthen  the  quality  of  independent  mail-in  reviews  by  making  them  an  offi¬ 
cial  requirement  rather  than  optional  and  by  attaching  more  importance  to  them 
(at  present,  the  community  believes  that  mail-in  reviews  are  not  considered  very 
important).  Second,  some  suggested  that  the  quality  of  the  mail-in  reviews  could 
be  made  uniformly  higher  if  the  mail-in  participants  knew  that  they  would  be  in¬ 
cluded  in  the  evaluation  feedback  provided  to  proposal  writers.  Currently,  the 
mail-in  reviews  are  not  published;  drey  are  merely  summarized  by  NASA.  Third, 
many  suggested  that  mail-in  reviews  would  be  better  if  proposal  writing  and 
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reviewing  workloads  were  reduced,  thereby  giving  mail-in  reviewers  more  time 
to  devote  to  these  evaluations. 
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Chapter  5 


Key  Survey  Results:  Proposal  Writers 


In  this  chapter,  we  discuss  the  key  results  from  our  survey  of  proposal  writ¬ 
ers.  We  summarize  the  overall  quality  of  the  science  grant  process  from  the 
point  of  view  of  proposal  writers,  present  their  responses  to  key  survey  ques¬ 
tions,  and  discuss  the  implications  of  those  responses  for  improving  the  science 
grant  process.  Appendix  I  shows  the  distribution  of  responses  for  each  survey 
question,  and  Appendix  J  provides  the  comments  made  by  the  respondents. 


Quality  Index 

From  the  survey  results,  we  computed  a  quality  index  to  show  how  well 
proposal  writers'  quality  standards  have  been  met  and  to  identify  those  compo¬ 
nents  of  quality  that  need  improvement.  We  calculated  that  proposal  writers 
gave  a  56  percent  average  rating,  on  a  0  percent  to  100  percent  scale,  for  the  over¬ 
all  quality  of  the  science  grant  process.  That  is,  proposal  writers  indicated  that 
56  percent  of  their  standards  were  met  with  the  current  science  grant  process. 
Table  5-1  shows  the  construction  of  the  proposal  writer  quality  index. 

The  satisfaction  scores  on  all  four  quality  components  for  proposal 
writers  —  proposal  guidance,  processing  time,  fairness,  and  feedback  —  are  rela¬ 
tively  low,  ranging  from  approximately  48  to  66  percent.  We  believe  that  any 

Table  5-1. 

Proposal  Writer  Quality  Index 


Quality  component 

Percentage  of 
standard3 

(i) 

Weight*1 

(2) 

Contribution  to  total 
(3)  =  (1)  x  (2) 

Proposal  guidance 

0.217 

0.142 

Processing  time 

0.596 

0.221 

0.132 

Fairness 

0.296 

0.160 

Feedback 

0.266 

0.127 

Total 

— 

1.000 

0.561 

4  Percentage  of  standard  is  computed  by  taking  the  average  actual  proposal  writer  scores  for  the  answers  to 
questions  that  fall  under  each  quality  component  —  scores  that  range  from  1 ,  “not  satisfied,”  to  a  maximum 
possible  score  of  5,  or  “extremely  satisfied”  —  relative  to  the  maximum  of  a  5  score,  the  standard  of  excel¬ 
lence. 

b  Respondents  used  1-to-5  scores  to  register  the  degree  of  importance  that  they  attached  to  each  quality 
component.  We  calculated  a  weight  for  each  quality  component  by  taking  its  total  importance  scores  relative 
to  the  grand  total  of  all  quality  component  scores. 
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quality  component  with  a  satisfaction  score  under  70  percent  is  a  target  for  im¬ 
provement.  We  now  provide  the  highlights  for  each  of  the  quality  components 
in  the  order  in  which  they  appear  in  Table  5-1. 


Proposal  Guidance 

Proposal  writers  gave  relatively  good  marks  to  the  guidance  they  received 
on  proposal  contents  and  on  preparation  and  submission,  but  somewhat  lower 
marks  to  the  guidance  on  program  scope,  costing,  and  evaluation  criteria. 
Figure  5-1  shows  the  average  satisfaction  scores  for  each  of  these  elements  of 
proposal  guidance.  Specifically,  proposal  writers  gave  proposal  contents  and 
preparation  and  submission  average  scores  of  3.5,  indicating  that  NASA  had 
achieved  70  percent  of  the  quality  standards  —  reasonably  good  results.  How¬ 
ever,  the  average  scores  for  the  other  elements  of  guidance  fell  below  the 
70  percent  level,  or  below  3.5  on  the  l-to-5  satisfaction  scale.  Program  scope  re¬ 
ceived  a  3.3  score;  costing,  a  3.1  score;  and  evaluation  factors,  a  3.0  score  —  all  at 
or  slightly  more  than  fairly  satisfied.1  Thus,  NASA  should  strengthen  its  overall 
proposal  guidance  by  improving  the  guidance  it  provides  on  program  scope, 
costing,  and  evaluation  criteria. 


Proposal  guideline 
Contents 

Preparation 

Scope 

Costing 

Evaluation 

1.0  1.5  2.0  2.5  3.0  3.5  4.0  4.5  5.0 

Average  satisfaction  level 


Figure  5-1 . 

Satisfaction  with  Proposal  Writing  Guidance 


1  We  stratified  the  sample  of  proposal  writers  by  those  who  were  successful  and  those 
who  were  less  successful  in  receiving  grants  to  ensure  that  the  survey  statistics  for  pro¬ 
posal  guidance  (and  other  factors)  were  sound.  Such  stratification  was  necessary  because 
the  satisfaction  statistics  tend  to  be  higher  for  the  successful  proposal  writers  in  most 
categories  of  proposal  guidance  (except  preparation  and  submission,  where  there  was  no 
effect  of  the  degree  of  proposal  success  on  satisfaction  levels  whatsoever).  So,  with  our 
stratified  sampling  procedures,  the  average  statistics  reported  are  an  accurate  reflection 
of  the  views  of  the  population  of  proposal  writers  on  proposal  guidance  (and  other  fac¬ 
tors). 
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Proposal  writers  made  many  comments  on,  and  suggestions  for,  improving 
proposal  guidance.  (See  Appendix  J.)  We  provide  some  of  them  here,  catego¬ 
rized  by  proposal  guidance  element: 

♦  Contents  and  Preparation 

'The  NRA  is  poorly  organized  —  one  does  not  know  where  to  look  for  certain 
pieces  of  information." 

"To  understand  all  of  the  requirements  one  needs  to  flip  back  and  forth  be¬ 
tween  the  general  instructions  in  the  Appendices  and  the  specific  instructions 
in  the  body  of  the  [announcement]." 

"The  format  of  the  proposals  themselves  could  be  streamlined.  Eliminate  the 
redundant  and  pedantic  "Budget  Summary"  pages.  Obtain  a  single  annual 
copy  of  the  drug-free  workplace  [and  other  forms]  from  each  university  in¬ 
stead  of  the  wasteful  duplication  of  putting  them  in  every  proposal.  The  forms 
are  signed  by  a  representative  of  the  whole  university  . . ." 

"Send  out  self-contained  NRAs  that  contain  all  the  necessary  forms  . . ." 

"Give  a  sample  submission  . .  ." 

"A  summary  sheet  of  requirements  would  speed  the  preparation." 

♦  Program  Scope 

"The  true  [program]  scope  is  difficult  to  see.  Talking  to  NASA  sometimes 
clarifies  things  quite  a  lot." 

"Program  scope  described  in  NRAs  is  often  not  consistent  with  what  proposals 
are  actually  invited  or  funded." 

"The  areas  of  interest  should  be  clearly  stated  and  prioritized."  ' 

♦  Costing 

"NASA  (the  government)  must  abandon  the  requirement  that  funds  must  be 
used  within  [the]  year  [of  funding].  Costs  can  often  be  reduced  if  expenses  can 
be  postponed." 

"Writing  a  cost  management  plan  is  difficult  for  most  research  efforts,  in  which 
it  is  difficult  to  gauge  direction  more  than  a  few  weeks  at  a  time.  This  may  be 
best  reserved  for  large  project  team  proposals." 

"NASA  should  cap  allowable  indirect  costs  to:  40%  on  salaries  and  wages  and 
benefits;  15%  on  supplies,  travel,  publication  costs,  etc.;  0%  on  equipment,  stu¬ 
dent  stipends  and  tuition." 

♦  Evaluation  Criteria 

"Clear  evaluation  criteria  [are  needed]." 
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"I  do  not  like  the  nebulous  programmatic  factors." 

"'Relevance'  is  an  almost  worthless  evaluation  factor,  and  I  bet  that  most  of  the 
community  feels  that  way." 

"I  would  like  to  see  an  evaluation  option  for  each  of  the  specific  evaluation  fac¬ 
tors,  not  just  [for]  the  factors  [taken]  together." 

We  believe  that  NASA  can  easily  adjust  its  proposal  guidelines  to  improve 
proposal  writer  satisfaction  with  them.  Guidelines  on  contents  and  preparation 
could  be  streamlined,  organized  better,  and  simplified.  Program  scope  could  be 
more  specific  in  terms  of  priorities  and  maximum  funding  levels,  thereby  help¬ 
ing  to  avoid  unnecessary  proposal  writing  and  reviewing.  The  policies  and  pro¬ 
cedures  for  both  costing  and  evaluation  criteria  could  be  better  explained  and 
revised,  as  may  be  necessary. 


Processing  Time 

The  proposal  writers  are  satisfied  with  the  current  standards  for  preparation 
time  and  did  not  suggest  changing  them.  However,  they  are  not  nearly  as  satis¬ 
fied  with  the  amount  of  time  for  obtaining  feedback  and  funding.  Table  5-2 
shows  the  time  standards  (both  current  and  desired)  for  processing  proposals,  as 
well  as  the  actual  times.  All  times  are  expressed  in  months. 

Table  5-2. 

Proposal  Processing  Times 
(months) 


Processing  element 

Time  standard 

Actual  time 

Current 

Desired 

Preparation 

3 

3 

Feedback 

6 

4 

Funding 

_ a 

2 

Total 

— 

9 

13 

a  NASA  has  no  time  standard  for  disbursing  funds. 


Proposal  writers  were  not  very  satisfied  with  the  time  for  obtaining  feedback 
after  submitting  their  proposals.  Although  NASA  has  met  its  current  six-month 
feedback  standard  most  of  the  time,  proposal  writers  would  like  NASA  to  estab¬ 
lish  a  more  ambitious  time  standard. 

NASA  has  no  time  standard  for  issuing  funds  after  it  has  awarded  a  grant, 
but  the  time  has  averaged  about  four  months.  Proposal  writers  were  very  dis¬ 
turbed  about  such  delays.  They  would  like  to  see  a  two-month  (maximum) 
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standard  adopted  for  receiving  funding  after  they  have  been  notified  of  receiving 
the  grant  award. 

The  following  comments  on  this  issue  are  a  sample  of  those  provided  in  Appen¬ 
dix  J: 

"Once  an  award  is  made,  the  delay  in  actually  getting  the  money  to  an  institu¬ 
tion  has  crept  up  to  an  additional  2-3  months." 

"A  recurrent  problem  has  been  the  length  of  time  between  notification  of  re¬ 
view  results  and  actual  funding  (ranging  from  4  to  6  months)." 

". . .  the  response  time  [for  receiving  funds]  is  often  6-9  months." 

"Numerous  phone  calls  from  us  are  required  in  order  to  receive  payment.  We 
are  often  told  that  the  grant  papers  are  'lost'  in  huge  stacks  of  paperwork  on 
various  individual  desks." 

We  believe  that  NASA  should  tighten  its  processing  times  to  a  total  of  nine 
months  from  the  point  of  proposal  preparation  to  receipt  of  funding.  These  stan¬ 
dards  should  be  provided  to  the  National  Performance  Review  and  published  in 
NASA's  Research  Announcements.  Process  improvements,  especially  with  the 
time  for  issuing  funding  after  notification  of  award,  will  be  needed  to  reach  these 
stricter  standards.  The  Office  of  Space  Science  should  make  the  necessary  im¬ 
provements  to  meet  these  new  standards.2 


Fairness 


Proposal  writers  were  not  too  satisfied  with  the  fairness  of  the  evaluations  of 
their  proposals.  They  gave  an  average  score  of  2.7  or  slightly  to  fairly  satisfied 
on  the  l-to-5  satisfaction  scale.  Figure  5-2  shows  the  distribution  of  proposal 
writers'  responses  on  this  question.  Whereas  30  percent  of  the  survey  respon¬ 
dents  were  fairly  satisfied  (gave  a  3  score)  and  27  percent  were  more  than  fairly 
satisfied  (gave  either  a  4  or  5  score),  42  percent  were  less  than  fairly  satisfied 
(gave  either  a  1  or  2  score).  We  found  no  statistically  significant  differences  in 
proposal  writers'  assessments  of  review  fairness  according  to  either  their  scien¬ 
tific  discipline  or  years  of  experience  in  submitting  proposals.  However,  the  dif¬ 
ferences  in  assessment  of  review  fairness  according  to  proposal  success  were 
statistically  significant;  successful  proposal  writers  tended  to  view  the  process  as 
fair,  while  less  successful  proposal  writers  were  less  satisfied  that  the  process  is 
fair. 


2  This  office  should  also  benchmark  other  Federal  grant  processes  and  make  further 
improvements  as  necessary. 
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Percent 


LEGEND 


1  =  Not  Satisfied 

2  =  Slightly  Satisfied 

3  =  Fairly  Satisfied 

4  =  Very  Satisfied 

5  =  Extremely  Satisfied 


Figure  5-2. 

Satisfaction  with  Review  Fairness:  Proposal  Writers 


Many  of  the  proposal  writers'  comments  and  suggestions  on  the  fairness  of 
proposal  evaluations  are  similar  to  those  of  the  peer  review  panelists.  Some  of 
the  proposal  writers'  comments  (from  Appendix  J)  are  quoted  here: 

"When  reviewing  proposals  for  scientific  merit,  do  not  include  the  investiga¬ 
tors'  names  or  institutions.  I  do  believe  that  there  is  a  great  bias  towards  those 
from  well-known  institutions.  I  have  experienced  this  when  I  moved  from  an 
institution  more  well-known  than  where  I  am  now.  My  science  efforts  have 
not  changed,  but  I  have  noticed  an  increase  in  difficulty  in  being  awarded 
grants." 

"I  very  strongly  support  the  panel  review  process  which  NASA  currently  uses 
to  evaluate  proposals.  [This]  process  is  much  fairer  (and  therefore  more  effec¬ 
tive)  than  the  reliance  on  remote  peer-reviewers  . . 

"Prefer  independent  individual  reviews  over  panel  [reviews].  Panel  [is]  sub¬ 
ject  to  politicking  by  one  dominant  [panel]  member." 

"Its  generally  a  good  process,  but  at  times  it  appears  exceedingly  difficult  for 
new  research  ideas  and  people  to  break  into  an  area." 

"It  is  simply  not  possible  to  have  a  fair  and  thoughtful  review  if  a  panelist  is  as¬ 
signed  60  to  70  proposals,  and  if  the  formal  meetings  are  limited  to  at  best 
2  days." 

We  believe  that  NASA  should  improve  the  fairness  of  the  evaluation  process 
as  perceived  by  proposal  writers.  NASA  could  make  mail-in  reviews  a  manda¬ 
tory  part  of  the  review  process  in  order  to  broaden  the  evaluation  inputs.  NASA 
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could  further  bolster  the  perception  of  fairness  by  sending  proposals  to  the  re¬ 
view  panel  and  independent  reviewers  without  the  identification  of  the  investi¬ 
gators  or  their  institutional  affiliations.  Also,  NASA  could  provide  further 
explanations  of  final  selections  after  panel  reviews.  Finally,  by  reducing  work¬ 
loads,  NASA  will  also  help  to  improve  the  thoroughness  and  fairness  of  the  re¬ 
view  process. 


Feedback 


Proposal  writers  felt  that  the  feedback  they  received  on  their  proposals  was 
only  slightly  to  fairly  adequate.  On  the  l-to-5  satisfaction  scale,  they  gave  an  av¬ 
erage  score  of  2.4  on  this  question.  Figure  5-3  shows  the  distribution  of  their  re¬ 
sponses  on  the  adequacy  of  evaluation  feedback.  The  answers  to  this  question 
were  universal:  We  found  no  statistically  significantly  differences  in  evaluation 
feedback  by  scientific  discipline  of  the  proposal  writers,  by  years  of  experience 
for  submitting  proposals,  or  by  success  in  receiving  grants.  Clearly,  the  proposal 
writers  as  a  whole  view  feedback  as  a  serious  problem. 


Percent 


Level  of  adequacy 


LEGEND 


1  -  Not  Adequate 

2  =  Slightly  Adequate 

3  =  Fairly  Adequate 

4  =  Very  Adequate 

5  =  Extremely  Adequate 


Figure  5-3. 

Satisfaction  with  Evaluation  Feedback 


Proposal  writers  provided  many  comments  and  suggestions  for  improving 
evaluation  feedback  (see  Appendix  J).  We  quote  of  a  few  of  their  concerns  here: 

"Generally,  more  openness  on  the  part  of  NASA  is  needed." 
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"Specific  written  comments  justifying  the  denial  of  a  proposal  by  the  NASA 
administrator  for  selection  are  essential  feedback . . .  The  step  between  the  re¬ 
view  panel  and  final  selection  seems  very  mystical!" 

"Too  often  the  final  reviews  are  watered  down  and  do  not  reflect  the  full  range 
of  issues  which  were  brought  up  in  the  panel  sessions.  [Investigators  should 
receive]  a  more  detailed  panel  report  (which  should  be  closer  to  a  discussion 
summary  than  a  "consensus"), . . .  complete  mail-in  reviews,  as  well  as  initial 
reports  of  the  principal  and  secondary  reviewers." 

"In  the  past  2  to  3  years,  NASA  Program  Managers  have  not  been  providing 
extensive  review  comments  on  proposals.  This  is  a  major  method  of  improv¬ 
ing  one's  science,  since  these  comments  are  often  valuable." 

We  believe  that  NASA  should  strike  a  better  balance  between  preparing  a 
general  summary  of  evaluation  results  and  providing  proposal  writers  with 
more  detailed  evaluation  information.  Evaluation  summaries  are  useful  for  justi¬ 
fying  funding  decisions,  while  more  detailed  evaluations  may  help  proposal 
writers  to  prepare  higher  quality  grant  proposals.  We  do  not  believe  that  these 
two  requirements  are  incompatible. 

NASA  should  consider  three  alternatives  for  improving  the  feedback  on  the 
evaluation  of  proposals.  One  alternative  is  to  provide  a  written  list  of  significant 
proposal  weaknesses  or  deficiencies  along  with  the  current  summary  of  evalua¬ 
tion  factors.  A  second  alternative  is  to  offer  a  face-to-face  debriefing  of  those 
weaknesses  or  deficiencies,  upon  written  request,  without  revealing  either  the 
technical  standing  of  competitors,  evaluation  scores,  or  information  that  is  not  re¬ 
leasable  under  the  Freedom  of  Information  Act,  such  as  proprietary  ideas.  The 
third  alternative  incorporates  both  of  these  two  alternatives:  Provide  a  written 
list  of  areas  of  weakness  or  deficiency  and  offer  a  debriefing  on  this  material.3 


3  Each  of  these  approaches  have  been  used  in  the  management  of  Federal  grants.  For 
example,  the  Public  Health  Service's  grants  policy  includes  both  written  comments  on 
proposal  deficiencies  as  well  as  an  offer  to  debrief  proposal  writers. 
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Appendix  A 


Executive  Order  12862, 
Setting  Customer  Service  Standards 


Executive  Order  12862, 

Setting  Customer  Service  Standards 


Executive  Order  12862  was  issued  in  two  phases.  Phase  I,  issued  on  Sept.  11, 
1993,  focuses  on  the  requirement  to  establish  customer  service  standards,  to  con¬ 
duct  customer  satisfaction  surveys,  and  to  develop  customer  service  plans. 
Phase  II,  issued  on  March  22, 1995,  provides  further  directions  to  agencies  for  im¬ 
proving  customer  service.  Both  documents  are  shown  in  their  entirety  in  this  ap¬ 
pendix. 
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Executive  Order  12862 
Phase  I  —  September  11, 1993 


Putting  people  first  means  ensuring  that  the  Federal  Government  provides 
the  highest  quality  service  possible  to  the  American  people.  Public  officials  must 
embark  upon  a  revolution  within  the  Federal  Government  to  change  the  way  it 
does  business.  This  will  require  continual  reform  of  the  executive  branch's  man¬ 
agement  practices  and  operations  to  provide  service  to  the  public  that  matches  or 
exceeds  the  best  service  available  in  the  private  sector. 

Now,  therefore,  to  establish  and  implement  customer  service  standards  to 
guide  the  operations  of  the  executive  branch,  and  by  the  authority  vested  in  me 
as  President  by  the  Constitution  and  the  laws  of  the  United  States,  it  is  hereby  or¬ 
dered: 

Section  1.  Customer  Service  Standards.  In  order  to  carry  out  the  principles  of 
the  National  Performance  Review,  the  Federal  Government  must  be  customer- 
driven.  The  standard  of  quality  for  services  provided  to  the  public  shall  be:  Cus¬ 
tomer  service  equal  to  the  best  in  business.  For  the  purposes  of  this  order,  "cus¬ 
tomer"  shall  mean  an  individual  or  entity  who  is  directly  served  by  a  department 
or  agency.  "Best  in  business"  shall  mean  the  highest  quality  of  service  delivered 
to  customers  by  private  organizations  providing  a  comparable  or  analogous 
service. 

All  executive  departments  and  agencies  (hereinafter  referred  to  collectively 
as  "agency"  or  "agencies")  that  provide  significant  services  directly  to  the  public 
shall  provide  those  services  in  a  manner  that  seeks  to  meet  the  customer  service 
standard  established  herein  and  shall  take  the  following  actions: 

(a)  identify  the  customers  who  are,  or  should  be,  served  by  the  agency; 

(b)  survey  customers  to  determine  the  kind  and  quality  of  services  they  want 
and  their  level  of  satisfaction  with  existing  services; 

(c)  post  service  standards  and  measure  results  against  them; 

(d)  benchmark  customer  service  performance  against  the  best  in  business; 

(e)  survey  front-line  employees  on  barriers  to,  and  ideas  for,  matching  the  best 
in  business; 

(f)  provide  customers  with  choices  in  both  the  sources  of  service  and  the  means 
of  delivery; 

(g)  make  information,  services,  and  complaint  systems  easily  accessible;  and 

(h)  provide  means  to  address  customer  complaints. 

Section  2.  Report  on  Customer  Service  Surveys.  By  March  8,  1994,  each  agency 
subject  to  this  order  shall  report  on  its  customer  surveys  to  the  President.  As  in¬ 
formation  about  customer  satisfaction  becomes  available,  each  agency  shall  use 
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that  information  in  judging  the  performance  of  agency  management  and  in  mak¬ 
ing  resource  allocations. 

Section  3.  Customer  Service  Plans.  By  September  8, 1994,  each  agency  subject  to 
this  order  shall  publish  a  customer  service  plan  that  can  be  readily  understood 
by  its  customers.  The  plan  shall  include  customer  service  standards  and  describe 
future  plans  for  customer  surveys.  It  also  shall  identify  the  private-  and  public- 
sector  standards  that  the  agency  used  to  benchmark  its  performance  against  the 
best  in  business.  In  connection  with  the  plan,  each  agency  is  encouraged  to  pro¬ 
vide  training  resources  for  programs  needed  by  employees  who  directly  serve 
customers  and  by  managers  making  use  of  customer  survey  information  to  pro¬ 
mote  the  principles  and  objectives  contained  herein. 

Section  4.  Independent  Agencies.  Independent  agencies  are  requested  to  adhere 
to  this  order. 

Section  5.  Judicial  Review.  This  order  is  for  the  internal  management  of  the  ex¬ 
ecutive  branch  and  does  not  create  any  right  or  benefit,  substantive  or  proce¬ 
dural,  enforceable  by  a  party  against  the  United  States,  its  agencies  or 
instrumentalities,  its  officers  or  employees,  or  any  other  person. 

WILLIAM  J.  CLINTON 

THE  WHITE  HOUSE 
September  11, 1993 
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Executive  Order  12862 
Phase  II  —  March  22, 1995 


THE  WHITE  HOUSE 

WASH  I  NGTON 

March  22,  1995 


MEMORANDUM 

SUBJECT: 


FOR  HEADS  OF  EXECUTIVE  DEPARTMENTS  AND  AGENCIES 
Improving  Customer  Service 


In  the  first  phase  of  this  Administration's  reinventing 
government  initiative ,  I  established  the  principle  that 
government  must  be  customer -driven.  Executive  Order 
No.  12862,  "Setting  Customer  Service  Standards,"  called 
for  a  revolution  within  the  Federal  Government  to  change 
the  way  it  does  business .  The  initial  agency  responses 
to  that  order,  including  the  service  standards  published 
m  September  1994,  have  begun  the  process  of  establishing 
a  more  customer- focused  government.  For  the  first  time 
the  Federal  Government's  customers  have  been  told  what 
they  have  a  right  to  expect  when  they  ask  for  service. 

In  the  second  phase  of  reinventing  government  ("Phase  II"), 
tins  effort  should  be  continued  and  integrated  with  other 
restructuring  activities .  The  first  question  agencv 
restructuring  teams  should  ask  is  whether  a  program  or 
function  is  critical  to  the  agency' s  missions  based  on 
"customer"  input.  To  carry  out  this  Phase  II  effort 
and  assure  that  government  puts  the  customer  first,  I 
am  now  directing  the  additional  steps  set  forth  in  this 
memorandum . 


Actions .  The  agencies  covered  by  Executive  Order 
No.  12862  are  directed  as  follows: 


order  to  continue  customer  service  reform,  agencies 
shall  treat  the  requirements  of  Executive  Order  No.  12862  as 
continuing  requirements .  The  actions  the  order  prescribes ,  such 

aS  customers '  surveying  employees,  and  benchmarking, 

shall  be  continuing  agency  activities.  The  purpose  of  these 
actions  will  remain  as  indicated  in  Executive  Order  No.  12862  -- 
the  establishment  and  implementation  of  customer  service 
standards  to  guide  the  operations  of  the  executive  branch. 


2.  Agencies  shall,  by  September  1,  1995,  complete  the 
publication  of  customer  service  standards,  in  a  form  readily 
available  to  customers,  for  all  operations  that  deliver 
significant  services  directly  to  the  public.  This  shall  include 
services  that  are  delivered  in  partnership  with  State  and  local 
governments,  services  delivered  by  small  agencies  and  regulatory 
agencies,  and  customer  services  of  enforcement  agencies 
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3.  Agencies  shall,  on  an  ongoing  basis,  measure  results 
achieved  against  the  customer  service  standards  and  report  those 
results  to  customers  at  least  annually.  Reports  should  be  in 
terms  readily  understood  by  individual  customers.  Public 
reports  shall  be  made  beginning  no  later  than  September  15, 

1995.  Measurement  systems  should  include  objective  measures 
wherever  possible,  but  should  also  include  customer  satisfaction 
as  a  measure.  Customer  views  should  be  obtained  to  determine 
whether  standards . have  been  set  on  what  matters  most  to  the 
customer.  Agencies  should  publish  replacement  standards  if 
needed  to  reflect  these  views. 

4.  Development  and  tracking  of  customer  service  measures, 
standards,  and  performance  should  be  integrated  with  other 
performance  initiatives,  including  Phase  II  restructuring. 
Customer  service  standards  also  should  be  related  to  legislative 
activities,  including  strategic  planning  and  performance 
measurement  under  the  Government  Performance  and  Results  Act 

of  1993 ,  reporting  on  f inancial  and  program  performance  under 
the  Chief  Financial  Officers  Act  of  1990,  and  the  Government 
Management  and  Reform  Act  of  1994.  Operating  plans,  regulations 
and  guidelines,  training  programs,  and  personnel  classification 
and  evaluation  systems  should  be  aligned  with  a  customer  focus. 

5 .  Agencies  shall  continue  to  survey  employees  on  ideas 
to  improve  customer  service,  take  action  to  motivate  and 
recognize  employees  for  meeting  or  exceeding  customer  service 
standards,  and  for  promoting  customer  service.  Without 
satisfied  employees,  we  cannot  have  satisfied  customers. 

6 .  Agencies  should  initiate  and  support  actions  that  cut 
across  agency  lines  to  serve  shared  customer  groups.  Agencies 
should  take  steps  to  develop  cross-agency,  one-stop  service  to 
customer  groups,  so  their  customers  do  not  needlessly  go  from 
one  agency  to  another.  Where  possible,  these  steps  should 
take  advantage  of  new  information  technology  tools  to  achieve 
results . 

The  standard  of  quality  we  seek  from  these  actions  and  the 
Executive  order  is  customer  service  for  the  American  people 
that  is  equal  to  the  best  in  business. 

Independent  Agencies.  Independent  agencies  are  requested 
to  adhere  to  this  directive. 

Judicial  Review.  This  directive  is  for  the  internal 
management  of  the  executive  branch  and  does  not  create  any 
right  or  benefit,  substantive  or  procedural,  enforceable 
by  a  party  against  the  United  States,  its  agencies  or 
instrumentalities,  its  officers  or  employees,  or  any  other 
person . 

OJL^tro,. 
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NASA  Request  for  Generic  Survey 
Clearance  Under  Executive  Order  12862, 
August  1994 

Introduction 

The  National  Aeronautics  and  Space  Administration  (NASA)  requests  that 
the  Office  of  Management  and  Budget  (OMB)  grant  it  a  generic  clearance  to 
undertake  customer  satisfaction  surveys  as  required  by  Executive  Order  12862, 
Setting  Customer  Service  Standards  (the  EO).  To  undertake  these  customer  satis¬ 
faction  surveys  properly,  we  have  carefully  reviewed  the  EO  and  its  require¬ 
ments,  have  regularly  sought  EO  clarification  from  the  staff  of  the  National 
Performance  Review  (NPR),  where  necessary,  and  have  thoroughly  examined 
OMB's  Resource  Manual  for  Customer  Surveys.  To  help  design,  conduct,  and  ana¬ 
lyze  the  customer  satisfaction  surveys,  we  have  also  retained  professional  statis¬ 
tical  personnel  who  have  attended  the  1994  OMB-sponsored  course.  Joint 
Program  in  Survey  Methodology,  to  review  OMB  statistical  survey  requirements. 
We  believe  that  NASA  is  in  compliance  with  all  EO  requirements. 

For  purposes  of  this  generic  clearance,  NASA  requests  approval  to  take  pe¬ 
riodic  EO  customer  satisfaction  surveys  of  its  nongovernmental  direct  customers. 
Specific  examples  of  NASA's  nongovernmental  direct  customers  include  univer¬ 
sity  scientists  who  apply  for  NASA  science  grants;  industrial  firms  that  utilize 
NASA  wind  tunnels  to  conduct  aeronautical  research;  educational  institutions 
that  utilize  NASA  scientific  materials  to  further  the  educational  process;  and  U.S. 
industry  and  academic  institutions  that  utilize  engineering  and  technology  infor¬ 
mation  disseminated  by  NASA  to  facilitate  private  innovation.  Current  custom¬ 
ers,  not  potential  ones,  will  be  surveyed  for  the  EO.  They  will  be  surveyed 
annually  but  only  on  a  voluntary  basis,  and  they  will  be  asked  both  quantitative 
and  qualitative  questions. 

The  EO  customer  surveys  will  provide  NASA  with  customer  satisfaction  in¬ 
formation  that  it  does  not  now  possess.  No  sensitive  information  will  be  col¬ 
lected. 

The  total  survey  burden  on  NASA's  non-governmental  direct  customers  will 
be  minimal  because  there  are  relatively  few  such  customers  in  each  area  (gener¬ 
ally  the  population  is  1,000  or  less  for  each).  We  anticipate  conducting  10  mail 
surveys,  20  focus  groups,  and  10  comment-card  efforts  per  year  for  each  of  the 
next  three  years.  The  mail  surveys  will  have  a  customer  burden  of  about 
1,250  hours  per  year  (fifteen  minutes  per  customer  respondent  x  500  (sample 
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size)  respondents  per  mail  survey  x  10  mail  surveys).  Also  on  an  annual  basis, 
focus  group  burdens  will  approximate  2,400  hours  (8  hours  per  focus  group 
member  x  15  members  per  focus  group  x  20  focus  groups),  and  customer  bur¬ 
dens  for  comment  cards  will  be  about  1,250  hours  (quarter  of  an  hour  per  com¬ 
ment  card  respondent  x  500  (sample  size)  respondents  per  comment  card  effort  x 
10  comment  card  efforts).  For  tire  combined  mail  surveys,  focus  groups,  and 
comment  cards,  customer  burdens  will  not  exceed  4,900  hours  for  each  of  the 
next  three  years. 

NASA  agrees  to  provide  OMB  with  mail  surveys,  comment  cards,  and  focus 
group  scripts  as  they  are  conducted,  and  with  an  annual  progress  report  that  ex¬ 
plains  the  survey  activities  during  the  year,  their  results,  and  respondent  bur¬ 
dens. 

This  request  for  generic  clearance  is  organized  into  three  parts:  Approach  to 
EO,  Surveying  Customers,  and  Terms  of  Generic  Clearance.  Also,  we  attach  the 
"Supporting  Statement"  required  for  OMB  approval  under  the  Paperwork  Re¬ 
duction  Act  and  5  CFR 1320. 


Approach  to  EO 

EO  12862  requires  that  NASA  take  customer  satisfaction  surveys  to  obtain 
customer  feedback  for  improving  its  products  and  services.  It  also  requires 
NASA  to  search  worldwide  for  best  industry  and  public-sector  methods  and  in¬ 
corporate  them  into  current  practices  as  may  be  appropriate.  Finally  the  EO  re¬ 
quires  that  front-line  employees  be  surveyed  on  barriers  to,  and  ideas  for, 
matching  the  best  in  industry  or  in  the  public  sector.  We  will  first  focus  our  at¬ 
tention  on  taking  a  customer  satisfaction  survey  in  each  area.  This  will  give 
NASA  an  opportunity  to  better  understand  its  customers  and  evaluate  quality 
standards,  a  necessary  first  step  before  incorporation  of  best  industry  or  public- 
sector  standards  in  NASA  practices. 

Customer  satisfaction  surveys  will  be  designed  to  address  the  following 
steps  in  order  to  implement  the  EO  properly: 

1.  Quality  Standards:  Customers  evaluate  standards  for  key  quality  dimensions 
of  NASA's  products  or  services.  For  many  quality  dimensions,  NASA  has 
already  expressed  or  may  be  able  to  express  customer  standards  in  quantita¬ 
tive  terms.  However,  in  some  cases,  a  quality  dimension  may  be  purely 
qualitative  in  nature  and  customer  responses  will  help  determine  the  stan¬ 
dard  for  that  type  of  dimension.  In  any  case,  customers  will  be  given  the 
opportunity  to  express  their  feelings  about  customer  standards  using  a  five- 
point  scale  in  the  survey. 

2.  Performance  Feedback:  Customers  provide  feedback  on  the  extent  to  which 
customer  standards  are  actually  met.  Customer  satisfaction  scores  will  be 
provided  in  the  survey  to  gauge  the  extent  to  which  customers  are  satisfied 
with  delivery  quality. 
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3.  Performance  Gaps:  An  evaluation  is  made  of  the  gaps  between  NASA 
product/ service  standards  and  actual  delivery  levels.  Such  an  evaluation 
will  be  made  by  comparing  results  from  steps  (1)  and  (2). 

4.  Importance  of  Quality  Dimensions:  The  relative  importance  that  customers  at¬ 
tach  to  each  quality  dimension  of  NASA's  products  or  services.  Customer 
importance  scores  will  be  obtained  from  the  survey  and  used  to  make  this 
determination. 

5.  Overall  Quality  Performance:  An  assessment  is  made  of  NASA's  overall  qual¬ 
ity  performance.  Such  an  assessment  will  be  based  upon  the  construction  of 
an  overall  quality  index,  derivable  from  steps  (1)  through  (4)  above.  The 
overall  quality  index  summarizes  the  extent  to  which  NASA  has  been  satis¬ 
fying  its  customers,  on  the  average. 

6.  Implementation  Plan:  An  implementation  plan  is  drawn  up  to  prioritize  the 
areas  most  in  need  of  improvement.  The  components  of  the  quality  index 
can  be  used  to  show  the  effect  that  closing  each  delivery-standard  gap 
would  have  on  NASA's  overall  quality  performance.  With  such  informa¬ 
tion,  NASA  can  target  the  most  fruitful  avenues  for  improving  processes 
and  enhancing  performance. 

After  taking  the  surveys  and  going  through  steps  (1)  through  (6),  NASA  will 
then  utilize  its  overall  continual  improvement  program  —  including  benchmark¬ 
ing  and  front-line  employee  surveys  —  to  overcome  performance  deficiencies. 


Surveying  Customers 

In  OMB's  Resource  Manual  for  Customer  Surveys,  various  principles  are  de¬ 
lineated  to  design,  conduct,  and  analyze  customer  satisfaction  surveys. 

NASA  has  established  a  Headquarters  steering  group  on  customer  satisfac¬ 
tion  to  operate  as  an  independent  review  group.  That  steering  group  will  review 
the  scope  of  the  measurement  program,  identify  factors  and  characteristics  that 
contribute  to  customer  satisfaction,  target  customer  populations  for  study,  de¬ 
velop  information  on  how  to  contact  the  targeted  customers,  and  establish  proce¬ 
dures  for  avoiding  survey  duplication  effort.  Through  the  steering  group,  NASA 
has  already  identified  services  and  products,  described  customers  served,  devel¬ 
oped  candidate  areas  for  customer  standards,  and  compiled  a  list  of  potential 
customers  to  be  contacted.  Working  with  professional  statistical  personnel,  the 
steering  group  has  also  developed  a  performance  measurement  framework  for 
consistently  undertaking  and  gauging  improvement  initiatives  under  the  EO  and 
related  programs.  The  steering  group  has  also  facilitated  Agency  compliance 
with  the  EO  requirements  for  reporting  to  the  President:  the  March  8  Customer 
Service  Status  Report  and  the  September  8  Customer  Service  Plan.  Finally,  the  steer¬ 
ing  group  will  examine  each  survey  to  be  conducted,  to  prevent  internal  duplica¬ 
tion  and  unnecessary  burdens  on  customers. 
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For  mail  surveys,  we  have  adopted  the  Total  Design  Method  to  balance  sur¬ 
vey  costs  against  die  need  for  adequate  response  rates.  Survey  research  has 
shown  that  a  mail  survey,  by  itself,  is  often  not  sufficient  for  obtaining  an  ade¬ 
quate  customer  satisfaction  survey  response  rate,  usually  considered  to  be  at 
least  50  percent.  Initial  mail  surveys  are  relatively  inexpensive  but  they  often 
achieve  response  rates  considerably  below  50  percent.  We  plan  to  number  the 
questionnaires  to  keep  track  of  survey  responses.  To  increase  the  response  rate, 
we  have  established  procedures  to  undertake  additional  customer  contacts:  a 
post  card  alerting  would-be  respondents  of  the  upcoming  survey,  a  special  post 
card  reminding  would-be  respondents  to  respond  to  the  survey,  and  a  final  ap¬ 
peal  letter  and  additional  questionnaire  to  urge  response,  as  necessary.  Also,  we 
plan  to  design  each  mail  questionnaire  according  to  the  latest  survey  research  re¬ 
sults  to  facilitate  as  high  a  mail  response  rate  as  possible:  Include  cover  letter; 
make  questionnaire  answerable  within  15  minutes;  guarantee  confidentiality; 
permit  anonymous  responses;  avoid  open-ended  questions;  group  questions  by 
category;  and  present  the  questions  in  survey  booklet  form.  NASA  is  committed 
to  obtaining  high  response  rates,  and  we  will  designate  a  period  of  seven  full 
weeks  to  accomplish  the  Total  Design  Method  for  each  survey  effort. 

NASA  has  developed  three  surveys  that  are  in  the  process  of  being  pretested 
at  this  time.  Working  with  statistical  personnel,  NASA  has  designed  and  is  pre¬ 
testing  questionnaires  concerning  scientists  who  apply  for  science  grants,  indus¬ 
try  that  utilizes  NASA  wind  tunnels,  and  educational  institutions  that  avail 
themselves  of  NASA  science  programs.  We  anticipate  that  these  surveys  will  be 
fully  pretested  and  conducted  beginning  in  the  fall  of  1994.  Other  mail  survey 
plans,  focus  groups,  and  comment  cards  have  not  as  yet  been  made  final,  so  we 
cannot  report  on  them  at  this  time. 

Each  of  NASA's  direct  customer  populations  generally  is  considerably  be¬ 
low  6,000.  According  to  the  hypergeometric  distribution,  a  sample  of  500  or  a 
50  percent  sample  will  be  taken  for  populations  of  1,000  or  less  to  achieve  preci¬ 
sion  of  plus  or  minus  3  percent  at  the  95  percent  confidence  limit;  and  according 
to  the  normal  distribution,  maximum  samples  of  1,000  for  larger  populations. 
Stratified  sampling  will  not  be  necessary  most  of  the  time.  These  sampling  plans 
are  consistent  for  NASA  to  attain  a  high  response  rate. 

We  plan  to  enter  the  data  into  a  computer  and  double-check  their  accuracy. 
We  intend  to  utilize  the  Statistical  Analysis  System  (SAS)  to  manage  the  data 
base,  tabulate  the  data,  and  develop  special  tables  and  charts.  We  also  plan  to 
perform  multivariate  statistical  analyses  with  SAS  to  assess  whether  differences 
in  respondent  backgrounds  help  explain  differences  in  responses  to  a  question  or 
whether  responses  to  different  questions  are  related  to  one  another. 


Terms  of  Generic  Clearance 

NASA  is  requesting  generic  approval  to  conduct  a  number  of  customer 
satisfaction  surveys  from  FY94  through  FY97.  Specifically,  this  generic  request 
covers  surveys  of  NASA's  non-governmental  direct  customers,  including  pre- 
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tests.  Customer  burdens  will  be  about  4,900  hours  for  each  of  three  years  —  con¬ 
sisting  of  1,250  hours  from  mail  surveys,  2,400  hours  from  focus  groups,  and 
1,250  hours  from  comment  cards.  This  request  for  generic  clearance  does  not  in¬ 
clude  any  industry  benchmarking,  employee  surveys,  or  indirect  customer  sur¬ 
veys  that  NASA  may  take  in  connection  with  the  EO.  A  separate  request  would 
follow  for  such  surveys. 

NASA  agrees  to  follow  OMB's  Resource  Manual  for  Customer  Surveys  in  de¬ 
signing,  conducting  and  analyzing  customer  surveys.  Also,  NASA  agrees  to  fol¬ 
low  policies  and  procedures  in  the  1994  OMB-sponsored  course,  Joint  Program  in 
Survey  Methodology. 

NASA  further  agrees  to  submit  a  copy  of  each  survey  instrument  or  focus 
group  script  to  OMB  at  the  time  they  are  conducted.  In  addition,  NASA  agrees 
to  submit  annual  progress  reports  to  OMB  at  the  end  of  each  year  starting  in  1994 
and  ending  in  1997,  as  required  by  the  Resource  Manual  for  Customer  Surveys. 
These  reports  will  include  the  survey  instruments  and  focus  group  scripts  util¬ 
ized  during  the  year,  a  summary  of  their  intent,  respondent  burdens,  and  the  re¬ 
sults  achieved.  At  the  time  of  the  annual  progress  reports,  any  revisions  to  this 
generic  clearance  that  might  be  necessary  will  also  be  submitted  for  approval. 


Supporting  Statement  Under  the  Paperwork 
Reduction  Act  and  5  CFR 1320 

Justification 

1.  Information  Necessary:  Executive  Order  12862,  Setting  Customer  Service  Stan¬ 
dards,  requires  that  NASA  systematically  survey  its  customers  "to  determine 
the  kind  and  quality  of  services  they  want  and  their  level  of  satisfaction  with 
existing  services..."  NASA  plans  to  conduct  mail  surveys,  focus  group  ses¬ 
sions,  and  comment  card  inquiries  in  order  to  meet  the  requirements  of  the 
Executive  order.  The  use  of  these  instruments  is  the  only  means  we  have  for 
understanding  if  our  services  meet  the  needs  of  our  customers. 

2.  Purposes  and  Consequences:  Customer  information  will  be  used  by  all  of 
NASA  to  help  improve  customer  services.  Without  this  information,  we 
will  have  limited  knowledge  of  the  customer  services  that  may  need  im¬ 
provement,  of  the  priorities  our  customers  place  on  potential  improvement 
efforts,  and  of  appropriate  service  improvement  plans  for  raising  customer 
satisfaction  levels. 

3.  Improved  Information  Technology:  NASA  is  unaware  of  any  current  informa¬ 
tion  technology  that  can  improve  or  replace  the  use  of  these  survey  efforts. 

4.  Duplication:  NASA  has  established  a  Headquarters  steering  group  to  review 
all  customer  satisfaction  survey  efforts  and  avoid  duplication.  As  a  result, 
there  will  be  no  duplication. 
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5.  Other  Information  Sources:  We  do  not  have  similar  information  from  other 
sources. 

6.  Minimize  Burden:  The  survey  instruments  and  focus  group  scripts  will  be 
brief  with  only  very  basic  information  requested  to  measure  customer  satis¬ 
faction.  To  minimize  mail  respondent  burdens,  we  have  formulated  ques¬ 
tions  simply  and  directly,  used  closed-ended  questions  (not  open-ended 
questions),  made  the  questionnaire  answerable  within  15  minutes,  grouped 
questions  into  categories  for  ease  of  customer  response,  and  pretested  the 
questionnaires  to  ensure  minimal  burdens. 

7.  Collection  Frequency:  A  less  than  annual  data  collection  frequency  would  se¬ 
riously  jeopardize  NASA's  efforts  to  enhance  customer  satisfaction.  NASA 
requires  annual  data  collection  to  obtain  timely  customer  feedback,  to  ad¬ 
dress  customer  concerns  promptly,  and  to  encourage  continuous  improve¬ 
ment  efforts. 

8.  Following  Guidelines:  We  see  no  reason  to  deviate  from  5  CFR  1320.6  guide¬ 
lines. 

9.  Outside  Consultation:  NASA  has  been  in  close  contact  with  industry,  educa¬ 
tional  institutions,  and  the  scientific  community  to  obtain  their  views  on  the 
availability,  disclosure,  and  reporting  of  customer  satisfaction  information. 
Also,  we  have  been  thoroughly  pretesting  survey  instruments  with  potential 
respondents  to  ensure  that  specific  data  requests  are  clear,  reasonable,  and 
free  of  undue  burdens. 

10.  Confidentiality:  We  have  assured  survey  respondents  that  their  individual 
responses  would  not  be  reported.  Response  aggregates  are  adequate  for 
complying  with  the  Executive  order  and  only  they  will  be  reported. 

11.  Sensitive  Information:  Not  applicable 

12.  Annual  Costs:  The  full  NASA  cost  for  10  mail  questionnaires,  20  focus 
groups,  and  10  comment  card  efforts  is  estimated  to  be  $1.8  million. 

13.  Respondent  Burdens:  We  estimate  that  the  annual  respondent  burden  is 
4,900  hours  from  all  sources:  mail  surveys,  focus  groups,  and  comment  card 
efforts.  The  mail  surveys  will  have  a  respondent  burden  of  1,250  hours 
(15  minutes  per  respondent  x  500  respondents  per  survey  x  10  mail  sur¬ 
veys);  focus  groups  will  have  2,400  hours  (8  hours  per  focus  group  member 
x  15  members  per  focus  group  x  20  focus  groups);  and  comment  cards  will 
have  1,250  hours  (quarter  of  an  hour  per  respondent  x  500  respondents  per 
comment  card  effort  x  10  comment  card  efforts). 

14.  Burden  Changes:  Not  applicable 
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15.  Publishing:  Aggregate  survey  information  will  be  published  to  meet  the  re¬ 
quirements  of  the  Executive  order.  A  series  of  reports  will  be  issued  from 
FY95  through  FY97  as  survey  efforts  are  designed,  conducted,  and  analyzed. 


Collection  of  Information  Employing  Statistical  Methods 

1.  Sampling:  NASA  will  survey  approximately  10  industry,  25  science,  and 
5  educational  direct  customer  areas.  Each  of  these  areas  has  a  population  of 
no  more  than  6,000.  NASA  will  take  a  sample  of  no  more  than  1,000  to 
achieve  plus  or  minus  3  percent  precision  at  the  95  percent  confidence  level. 
Stratified  sampling  will  normally  not  be  necessary.  We  will  work  with  the 
Total  Design  Method  to  achieve  a  high  percent  response  rate,  targeted  at 
75  percent. 

2.  Collection  Procedures:  NASA  will  use  systematic  random  sampling  (selecting 
every  nth  one  from  the  population)  to  obtain  representative  survey  respon¬ 
dents.  Stratified  sampling  normally  would  not  be  necessary.  These  proce¬ 
dures  will  be  repeated  each  year.  And  we  do  not  anticipate  that  these 
survey  procedures  will  present  any  unusual  problems  or  respondent  bur¬ 
dens. 

3.  Maximize  Response  Rates:  We  have  taken  two  basic  steps  to  help  obtain  high 
response  rates.  First,  we  have  designed  the  mail  survey  instrument  in  such 
a  way  as  to  encourage  high  response  rates:  include  cover  letter,  answerable 
within  15  minutes,  guarantee  confidentiality,  avoid  open-ended  questions, 
group  questions  by  category,  and  present  the  questions  and  questionnaire  in 
an  appealing  survey  booklet  form.  Second,  we  have  adopted  the  Total  De¬ 
sign  Method  to  ensure  high  response  rates:  send  alert  post  cards  in  advance 
of  the  questionnaire;  send  mail  surveys  with  cover  letters,  send  special  re¬ 
minder  cards  to  non-respondents,  follow-up  with  a  final  appeal  letter  and 
additional  questionnaire,  as  necessary.  We  have  designated  a  full  seven 
weeks  to  accomplish  the  Total  Design  Method. 

4.  Tests:  We  will  pretest  each  survey  instrument  or  focus  group  script  to  elimi¬ 
nate  material  that  is  long,  awkwardly  worded,  ambiguous,  redundant,  or 
may  offend  or  sound  foolish  to  respondents.  Also,  we  will  use  the  pre-tests 
to  add  material  that  the  representative  respondents  feel  strongly  should  be 
included. 

5.  Consulting  Advice:  We  have  used  the  services  of  the  Logistics  Management 
Institute  (LMI)  to  help  design,  conduct,  and  analyze  various  customer  satis¬ 
faction  surveys;  LMI  points  of  contact  are  Larry  Schwartz  and  Brian  Mansir, 
(703)  917-9800. 
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OMB  Generic  Survey 
Clearance  Approval 


NOTICE  OF  OFFICE  OF  MANAGEMENT  -NO  BUUGtT  ACTION 


TO:  SHIRLEY  PEIGARE  ACTION  DATE 

NATIONAL  AERONAUTICS  ANO  SPACE  AON. 

CODE  NTO-1  -  NASA  HEAOQUARTERS 
WASHINGTON*  OC  20546 

National  Aeronautics  and  Space  Administration  10/12/94 
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Appendix  D 


NASA  Science  Grant  Customer  Survey: 

Proposal  Writers 


NASA  SCIENCE  GRANT  CUSTOMER  SURVEY 

Complying  with  Executive  Order  12862 

This  survey  is  the  first  of  many  we  will  conduct  to  better  understand  how  proposal  writers 
feel  about  NASA’s  science  grant  process.  Your  support  will  help  NASA  improve  its  grant 
process.  Please  answer  all  of  our  questions.  If  you  wish  to  comment  on  any  questions  or 
qualify  your  answers,  please  feel  free  to  use  the  space  in  the  margins.  Your  comments  will  be 
read  and  taken  into  account. 

Thank  you  for  your  help. 


Proposal  Writers 


National  Aeronautics  and  Space  Administration 
Office  of  Space  Science 
Code  S-l 

Washington,  D.C.  20546-0001 
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TIME 


Q-l  How  satisfied  are  you  with  the  3-month  period  allotted  for  preparing  grant 
proposals? 

Circle  the  number  of  your  answer. 

1  Not  Satisfied 

2  Slighdy  Satisfied 

3  Fairly  Satisfied 

4  Very  Satisfied 

5  Extremely  Satisfied 

Q-2  How  satisfied  are  you  with  NASA’s  6-month  standard  for  providing  proposal 
feedback? 

Circle  the  number  of  your  answer. 

1  Not  Satisfied 

2  Slighdy  Satisfied 

3  Fairly  Satisfied 

4  Very  Satisfied 

5  Extremely  Satisfied 

Q-3  What  time  standard  should  NASA  use  for  providing  proposal  feedback? 

indicate  number  of  months.  _  Months 

Q-4  What  has  been  your  actual  time,  on  the  average,  for  receiving  proposal  feedback? 

Circle  the  number  of  your  answer. 

1  Less  than  4  Months 

2  4-5  Months 

3  6-7  Months 

4  8-9  Months 

5  More  than  9  Months 

Q-5  How  satisfied  are  you  with  the  length  of  the  grant  period  for  executing  research? 

Circle  the  number  of  your  answer. 

1  Not  Satisfied 

2  Slighdy  Satisfied 

3  Fairly  Satisfied 

4  Very  Satisfied 

5  Extremely  Satisfied 


Q-6  What  period  of  time  would  you  like  to  see  for  executing  research? 

Circle  the  number  of  your  answer. 


1 

2 

3 

4 

5 


One  Year 
Two  Years 
Three  Years 
Four  Years 
Five  Years  or  More 


GUIDELINES 


Q-7  How  satisfied  are  you  with  the  overall  clarity,  simplicity,  and  completeness  of  the 
NASA  Research  Announcement  (NRA)? 

Circle  the  number  of  your  answer. 

1  Not  Satisfied 

2  Slighdy  Satisfied 

3  Fairly  Satisfied 

4  Very  Satisfied 

5  Extremely  Satisfied 

Q-8  How  satisfied  are  you  with  key  instructions  covered  in  the  NRA? 

Circle  the  number  of  your  answer  for  each  instruction. 


Instruction 

Not 

Satisfied 

Slightly 

Satisfied 

Fairly 

Satisfied 

Very 

Satisfied 

Extremely 

Satisfied 

Program 

Scope 

1 

2 

3 

4 

5 

Proposal 

Contents 

1 

2 

3 

4 

5 

Preparation 
&  Submission 

1 

2 

3 

4 

5 

Evaluation 

Factors 

1 

2 

3 

4 

5 

Costing 

1 

2 

3 

4 

5 
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GENERAL 


Q~9  How  adequate  is  NASA  feedback  on  your  proposals? 

Circle  the  number  of  your  answer. 

1  Not  Adequate 

2  Slightly  Adequate 

3  Fairly  Adequate 

4  Very  Adequate 

5  Extremely  Adequate 

Q-l  0  How  satisfied  are  you  that  NASA  fairly  evaluates  your  research  proposals? 

Circle  the  number  of  your  answer. 

1  Not  Satisfied 

2  Slighdy  Satisfied 

3  Fairly  Satisfied 

4  Very  Satisfied 

5  Extremely  Satisfied 


Q-ll  Adequate  feedback,  fairness,  processing  time,  and  process  guidance  are  essential 
elements  of  the  NASA  science  grant  program.  How  important  to  you  is  NASA’s 
feedback,  fairness,  processing  time,  or  process  guidance? 

Circle  the  number  of  your  answer  for  each  process  element. 


Process 

Not 

Slightly 

Fairly 

Very 

Extremely 

Element  important 

Important 

Important 

Important 

Important 

Feedback 

i 

2 

3 

4 

5 

Fairness 

i 

2 

3 

4 

5 

Processing  Time 

i 

2 

3 

4 

5 

Process  Guidance 

i 

2 

3 

4 

5 
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0-12  How  satisfied  are  you  with  the  overall  NASA  science  grant  process? 

Circle  the  number  of  your  answer. 

1  Not  Satisfied 

2  Slightly  Satisfied 

3  Fairly  Satisfied 

4  Very  Satisfied 

5  Extremely  Satisfied 


Q-13  What  is  your  general  area  of  scientific  concern? 

Circle  the  number  of  your  answer. 

1  Solar  System  Exploration 

2  Astrophysics 

3  Space  Physics 

4  Other _ _ 

Please  specify  your  subdiscipline _ _ _ _ 

Q-14  How  many  years  have  you  submitted  NASA  grant  proposals? 

Circle  the  number  of  your  answer. 

1  1  Year 

2  2-4  Years 

3  5 -7 Years 

4  8-10  Years 

5  More  than  1 0  Years 

Q-15  What  percentage  of  your  proposal  submissions  have  been  selected? 

Circle  the  number  of  your  answer. 

1  Less  than  20  Percent 

2  21  -40  Percent 

3  41  -60  Percent 

4  61  - 80  Percent 

5  More  than  80  Percent 


COMMENTS 


Would  you  like  to  suggest  any  specific  changes  that  would  improve  NASAs  science  grant 
process?  If  so,  please  use  this  space  for  that  purpose.  Also,  any  comments  that  you  may  have 
for  improving  future  survey  efforts  to  understand  how  you  feel  about  the  NASA  grant  process 
will  be  appreciated,  either  in  this  space  or  in  a  separate  letter. 


Your  contribution  to  this  survey  is  greatly  appreciated.  If  you  would  like  a  summary  of  results, 
please  print  your  name  and  address  on  the  back  of  the  return  envelope  provided  ( not  on  this 
questionnaire).  We  will  see  that  you  get  it. 


OMB  No.  2700-0084/Expiration  Date:  10/31/95 
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Appendix  E 


NASA  Science  Grant  Customer  Survey: 

Peer  Review  Panelists 


NASA  SCIENCE  GRANT  CUSTOMER  SURVEY 

Complying  with  Executive  Order  12862 

This  survey  is  the  first  of  many  we  will  conduct  to  better  understand  how  peer  review 
panelists  feel  about  NASA’s  science  grant  process.  Your  support  will  help  NASA  improve  its 
grant  process.  Please  answer  all  of  our  questions.  If  you  wish  to  comment  on  any  questions 
or  qualify  your  answers,  please  feel  free  to  use  the  space  in  the  margins.  Your  comments  will 
be  read  and  taken  into  account. 

Thank  you  for  your  help. 


Peer  Review 
Panelists 


National  Aeronautics  and  Space  Administration 
Office  of  Space  Science 
Code  S-l 

Washington,  D.C.  20546-0001 
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TIME  or  WORKLOAD 


Q-1  How  suitable  is  the  time  allotted  to  you  for  reviewing  grant  proposals  before 
panel  meetings? 

Circle  the  number  of  your  answer. 

1  Much  Too  Short 

2  Slightly  Too  Short 

3  Just  Right 

4  Slightly  Too  Long 

5  Much  Too  Long 


Q-2  How  appropriate  is  the  time  allotted  to  panel  meetings  for  reviewing  grant 
proposals? 

Circle  the  number  of  your  answer. 

1  Much  Too  Short 

2  Slightly  Too  Short 

3  Just  Right 

4  Slighdy  Too  Long 

5  Much  Too  Long 


Q-3  Each  proposal  is  reviewed  by  a  lead  and  one  or  more  secondary  reviewers.  Those 
reviewers  explain  and  critique  the  proposal  to  the  entire  work  group  or  subpanel. 
Have  you  served  as  a  lead  or  secondary  proposal  reviewer  since  1990? 

Circle  the  number  of  your  answer. 


1  Yes  Go  to  question  4 

2  No  Go  to  question  5 


CM  When  you  last  served  as  a  lead  or  secondary  reviewer,  how  many  proposals  were  you 
responsible  for  reviewing? 

Circle  the  number  of  your  answer 

1  1-3 

2  4-6 

3  7-9 

4  10-12 

5  More  than  12 
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GUIDELINES 


Q-5  How  satisfied  are  you  with  NASA’s  rules  and  instructions  on  evaluation  criteria, 
evaluation  methods,  or  proposal  rating  categories? 

Circle  the  number  of  your  answer  for  each  rule. 


Rule 

Not 

Satisfied 

Slightly 

Satisfied 

Fairly 

Satisfied 

Very 

Satisfied 

Extremely 

Satisfied 

Evaluation 

Criteria 

i 

2 

3 

4 

5 

Evaluation 

Methods 

i 

2 

3 

4 

5 

Proposal 

Rating 

i 

2 

3 

4 

5 

SUPPORT 


Q-6  How  satisfied  are  you  with  NASA’s  support  for  the  peer  review  process? 

Circle  the  number  of  your  answer  for  each  support  element. 


Support 

Element 

Not 

Satisfied 

Slightly 

Satisfied 

Fairly 

Satisfied 

Very 

Satisfied 

Extremely 

Satisfied 

Coordination 

1 

2 

3 

4 

5 

Meeting 

Facilities 

1 

2 

3 

4 

5 

Travel 

Arrangements 

1 

2 

3 

4 

5 

Meals 

Supplied 

1 

2 

3 

4 

5 

Supplies 

1 

2 

3 

4 

5 

Telephone/ 

Fax 

1 

2 

3 

4 

5 

Personal 

Computers 

1 

2 

3 

4 

5 

Clerical 

1 

2 

3 

4 

5 
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0-7  Are  you  typically  reimbursed  for  per  diem,  travel,  or  lodging  expenses? 

Circle  the  number  of  your  answer. 

1  Yes  Csr*  Go  to  question  8 

2  No  Go  to  question  9 


Q-8  How  satisfied  are  you  with  NASA’s  expense  reimbursements? 

Circle  the  number  of  your  answer  for  each  expense  item. 


Expense 

Item 

Not 

Satisfied 

Slightly 

Satisfied 

Fairly 

Satisfied 

Very 

Satisfied 

Extremely 

Satisfied 

Per  Diem 

i 

2 

3 

4 

'5 

Travel 

i 

2 

3 

4 

5 

Lodging 

i 

2 

3 

4 

5 

Q-9  Should  NASA  consider  using  information  technology  for  its  science  grant  process? 

Circle  the  number  of  your  answer  for  each  element  of  the  process. 


Process 

Element 

Yes 

No 

Preparation 

i 

2 

Submission 

i 

2 

Evaluation 

i 

2 

Award 

i 

2 

Notification 

i 

2 
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GENERAL 


Q-10  How  satisfied  are  you  with  the  qualifications  of  fellow  panel  members  or  the 
quality  of  mail-in  reviews? 

Circle  the  number  of  your  onswer  for  each  quality  issue. 


Quality  Not 

Issue  Satisfied 

Slightly 

Satisfied 

Fairly 

Satisfied 

Very 

Satisfied 

Extremely 

Satisfied 

Quality  of 

Mail-in  Reviews  1 

2 

3 

4 

5 

Panel 

Qualifications  1 

2 

3 

4 

5 

Q-1 1  How  satisfied  are  you  that  the  NASA  grant  process  fairly  evaluates  each  proposal? 

Circle  the  number  of  your  answer. 

1  Not  Satisfied 

2  Slighdy  Satisfied 

3  Fairly  Satisfied 

4  Very  Satisfied 

5  Extremely  Satisfied 

Q-12  How  satisfied  are  you  with  NASAs  procedures  for  avoiding  conflicts  of  interest  or 
protecting  proprietary  ideas? 

Circle  the  number  of  your  answer  for  each  ethical  issue. 


Ethical 

Issue 

Not 

Satisfied 

Slightly 

Satisfied 

Fairly 

Satisfied 

Very 

Satisfied 

Extremely 

Satisfied 

Conflict  of 
Interest 

1 

2 

3 

4 

5 

Proprietary 

Ideas 

1 

2 

3 

4 

5 
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Q-13  Are  you  satisfied  that  NASA’s  science  grant  process  results  in  the  selection  of  the 
best  research  projects? 

Circle  the  number  of  your  answer. 

1  Not  Satisfied 

2  Slightly  Satisfied 

3  Fairly  Satisfied 

4  Very  Satisfied 

5  Extremely  Satisfied 


Q-14  Selection  of  the  best  research  projects  and  overall  satisfaction  with  NASA’s  science 
grant  process  are  outcomes  that  depend  upon  a  number  of  process  elements. 

How  important  to  those  outcomes  are  the  following  NASA  process  elements? 


Process  Not 

Element  Important 

Slightly 

Important 

Fairly 

Important 

Very 

Important 

Extremely 

Important 

Time/Workload 

i 

2 

3 

4 

5 

Guidelines 

i 

2 

3 

4 

5 

Panel  Support 

i 

2 

3 

4 

5 

Panel 

Qualifications 

i 

2 

3 

4 

5 

Quality  of 

Mail-in  Reviews 

i 

2 

3 

4 

5 

Review  Fairness 

i 

2 

3 

4 

5 

Ethical  Issues 

i 

2 

3 

4 

5 
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Q-15  How  satisfied  are  you  with  the  overall  NASA  science  grant  process? 

Circle  the  number  of  your  answer. 

1  Not  Satisfied 

2  Slightly  Satisfied 

3  Fairly  Satisfied 

4  Very  Satisfied 

5  Extremely  Satisfied 


BACKGROUND 


Q-16  What  is  your  general  area  of  scientific  concern? 

Circle  the  number  of  your  answer. 

1  Solar  System  Exploration 

2  Astrophysics 

3  Space  Physics 

4  Other _ 


Please  specify  your  subdiscipline 


Q-17  How  long  have  you  worked  in  this  area? 

Indicate  number  of  years.  _ Years 

Q-18  How  many  times  have  you  ever  participated  on  NASA  review  panels? 

Circle  the  number  of  your  answer. 


1  Once 

2  2-4  Times 

3  5-7  Times 

4  8-10  Times 

5  More  than  10  Times 
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COMMENTS 


Would  you  like  to  suggest  any  specific  changes  that  would  improve  NASA’s  science  grant 
process?  If  so,  please  use  this  space  for  that  purpose.  Also,  any  comments  that  you  may  have 
for  improving  future  survey  efforts  to  understand  how  you  feel  about  the  NASA  grant  process 
will  be  appreciated,  either  in  this  space  or  in  a  separate  letter. 


Your  contribution  to  this  survey  is  gready  appreciated.  If  you  would  like  a  summary  of  results, 
please  print  your  name  and  address  on  the  back  of  the  return  envelope  provided  (not  on  this 
questionnaire).  We  will  see  that  you  get  it. 


OMB  No.  2700-0084/Expiration  Date:  10/31/05 
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Appendix  F 


Survey  Letters  and  Post  Cards 


Survey  Letters  and  Post  Cards 


Appendix  F  contains  the  letters  and  post  cards  that  were  sent  to  the  peer  re¬ 
view  panelists  and  proposal  writers  in  the  survey  sample: 

♦  Figure  F-l.  Post  Card.  Alerted  survey  participants  that  they  would  receive  a 
questionnaire  in  about  a  week;  sent  Jan.  25, 1995. 

♦  Figure  F-2.  Cover  Letter.  Accompanied  first  mailing  of  surveys;  sent  Feb.  1, 
1995. 

♦  Figure  F-3.  Thank  You  and  Reminder  Post  Card.  Thanked  those  who  had  re¬ 
sponded  and  reminded  those  who  had  not  done  so  that  a  response  was  es¬ 
sential;  sent  Feb.  8, 1995. 

♦  Figure  F-4.  Follozv-up  Letter.  Accompanied  second  survey  instrument  and 
made  a  final  appeal  for  survey  responses;  sent  Feb.  24, 1995. 
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Dear  Colleague:  * 

NASA's  Office  of  Space  Science  is  conducting  a 
survey  to  help  improve  its  science  grant  process. 
As  a  member  of  the  science  community,  your  opinions 
on  these  matters  are  critical  to  the  success  of  this 
effort.  In  about  a  week,  you  will  receive  a 
questionnaire  to  record  your  inputs.  Thank  you  in 
advance  for  your  cooperation. 

Carrie  L.  Sorrels 
Chief  of  Staff  for 
Space  Science 


Figure  F-1. 

Post  Card 


National  Aeronautics  and 
Space  Administration 

Headquarters 

Washington,  DC  20546-0001 


Reply  to  Attn  of: 


si 


Dear  Colleague: 

The  NASA  Office  of  Space  Science  is  currently  evaluating  the 
effectiveness  of  its  grant  process.  A  high  quality  grant  process 
is  important  for  discovering  new  knowledge  and  for  advancing 
technology  in  space  science.  The  President's  Executive  Order 
12862,  which  requires  such  a  review,  recognizes  that  soliciting 
the  opinions  of  scientists  like  yourself  is  the  key  to  evaluating 
improving  the  grant  process. 

As  a  member  of  the  NASA  science  community,  you  are  one  of  a  small 
number  of  people  being  asked  to  give  opinions  on  these  matters. 
Your  name  was  drawn  in  a  random  sample  of  community  members  who 
participated  in  the  space  science  program  in  1993.  You  may  be 
assured  of  complete  confidentiality.  The  questionnaire  has  an 
identification  number  for  mailing  purposes  only.  In  order  that 
the  results  truly  represent  the  space  science  community,  it  is 
important  that  each  questionnaire  be  completed  and  returned 
promptly. 

The  general  results  of  this  research  will  be  made  available  to 
Government  officials,  members  of  Congress  and  interested 
scientists  and  citizens .  You  may  receive  those  results  by  writing 
"copy  of  results  requested"  on  the  back  of  the  return  envelope  and 
printing  your  name  and  address  below  it.  Please  do  not  put  this 
information  on  the  questionnaire  itself. 

I  would  be  most  happy  to  answer  any  of  your  questions  on  the 
evaluation  of  the  space  science  grant  process.  Please  feel  free 
to  contact  me,  at  (fax)  202/358-3095,  or  (telephone)  202/358-0846. 

Thank  you  for  your  assistance. 


Sincerely, 


Chief  of  Staff 


Figure  F-2. 

Cover  Letter 
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Dear  Colleague: 

Last  week,  we  sent  you  a  questionnaire  to  receive  your  opinion 
about  the  NASA  Office  of  Space  Science  grant  process.  If 
you  have  already  completed  and  returned  the  questionnaire 
to  us,  please  accept  our  sincere  thanks.  If  not,  please  do  so 
today.  Because  the  questionnaire  has  been  sent  to  a  small 
but  representative  sample  of  the  NASA  science  grant 
community,  your  input  is  essential  for  accurate  results. 

If  by  some  chance  you  did  not  receive  the  questionnaire,  or 
you  misplaced  it,  please  call  me  at  (202)  358-0846  right  now, 
and  I  will  mail  another  one  to  you  today. 

Carrie  L.  Sorrels 
Chief  of  Staff  for 
Space  Science 


Figure  F-3. 

Thank  You  and  Reminder  Post  Card 
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National  Aeronautics  and 
Space  Administration 

Headquarters 

Washington,  DC  20546-0001 


Reply  to  Attn  of:  S 1 


Dear  Colleague : 

About  3  weeks  ago  by  way  of  mail,  I  asked  for  your  opinion  on  the 
NASA  grant  process  for  the  Office  of  Space  Science  (OSS) .  As  of 
today,  we  have  not  received  your  completed  questionnaire. 

The  views  of  the  NASA  science  community  are  essential  for 
improving  the  grant  process.  We  value  your  opinions  for 
evaluating  the  current  process,  for  identifying  problem  areas,  and 
for  suggesting  specific  improvements. 

I  am  writing  you  again  because  each  questionnaire  is  extremely 
important  for  obtaining  sound  survey  results .  Your  name  was  drawn 
through  a  scientific  sampling  in  which  everyone  who  participated 
in  the  OSS  grant  process  in  fiscal  year  1993  had  an  equal  chance 
of  participating  in  the  survey.  For  our  survey  results  to  be 
truly  representative  of  all  community  opinions,  it  is  necessary 
that  each  person  in  the  sample  complete  the  questionnaire. 

I  will  be  happy  to  send  you  a  copy  of  the  results.  Place  your 
name,  address,  and  “copy  of  results  requested"  on  the  back  of  the 
return  envelope.  We  expect  to  have  the  results  ready  this  summer. 

In  the  event  that  your  questionnaire  has  been  misplaced,  I  have 
enclosed  a  replacement .  Your  contribution  to  this  survey  will  be 
greatly  appreciated.  Please  contact  me  if  you  have  any  questions. 

I  can  be  reached  via  (fax)  202/358-3095  or  (telephone)  358-0846. 


Sincerely, 


Chief  of  Staff 


Figure  F-4. 

Follow-up  Letter 
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Appendix  G 


Survey  Questions  and  Responses: 

Peer  Review  Panelists 


Survey  Questions  and  Responses: 
Peer  Review  Panelists 


Time  or  Workload 

Q-l  How  suitable  is  the  time  allotted  to  you  for  reviewing  grant  proposals  be¬ 
fore  panel  meetings? 


LEGEND: 

1  =  Much  Too  Short 

2  =  Slightly  Too  Short 

3  =  Just  Right 

4  =  Slightly  Too  Long 

5  =  Much  Too  Long 


Figure  G-1. 

Suitability  of  Time  Allotted  for  Prepanel  Review 
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Q-2  How  appropriate  is  the  time  allotted  to  panel  meetings  for  reviewing  grant 
proposals? 


LEGEND: 

1  =  Much  Too  Short 

2  =  Slightly  Too  Short 

3  =  Just  Right 

4  =  Slightly  Too  Long 

5  =  Much  Too  Long 


Figure  G-2. 

Suitability  of  Time  Allotted  for  Panel  Review 


Q-3  Each  proposal  is  reviewed  by  a  lead  and  one  or  more  secondary  reviewers. 
Those  reviewers  explain  and  critique  the  proposal  to  the  entire  work  group 
or  subpanel.  Have  you  served  as  a  lead  or  secondary  proposal  reviewer 
since  1990? 


Figure  G-3. 

Service  as  Lead/ Secondary  Reviewer 
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Q-4  When  you  last  served  as  a  lead  or  secondary  reviewer,  how  many  proposals 
were  you  responsible  for  reviewing? 


Proposal  review  workload  (number) 


Figure  G-4. 

Lead! Secondary  Review  Workload 


Guidelines 


Q-5  How  satisfied  are  you  with  NASA's  rules  and  instructions  on  evaluation 
criteria,  evaluation  methods,  or  proposal  rating  categories? 


LEGEND: 

1  =  Not  Satisfied 

2  =  Slightly  Satisfied 

3  =  Fairly  Satisfied 

4  =  Very  Satisfied 

5  =  Extremely  Satisfied 


Level  of  satisfaction 


Figure  G-5. 

Satisfaction  with  Evaluation  Criteria 
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Level  of  satisfaction 


LEGEND: 

1  =  Not  Satisfied 

2  =  Slightly  Satisfied 

3  =  Fairly  Satisfied 

4  =  Very  Satisfied 

5  =  Extremely  Satisfied 


Figure  G-6. 

Satisfaction  with  Evaluation  Methods 


LEGEND: 

1  =  Not  Satisfied 

2  =  Slightly  Satisfied 

3  =  Fairly  Satisfied 

4  =  Very  Satisfied 

5  =  Extremely  Satisfied 


Level  of  satisfaction 


Figure  G-7. 

Satisfaction  with  Proposal  Rating 
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Support 


Q-6  How  satisfied  are  you  with  NASA's  support  for  the  peer  review  process? 


Level  of  satisfaction 


Figure  G-8. 

Satisfaction  with  Coordination 


Level  of  satisfaction 


Figure  G-9. 

Satisfaction  with  Meeting  Facilities 
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Percent 
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2  3  4 

Level  of  satisfaction 


26.63% 


LEGEND: 


1  =  Not  Satisfied 

2  =  Slightly  Satisfied 

3  =  Fairly  Satisfied 

4  =  Very  Satisfied 

5  =  Extremely  Satisfied 


Figure  G-10. 

Satisfaction  with  Travel  Arrangements 


Figure  G-11. 

Satisfaction  with  Meals  Supplied 
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Level  of  satisfaction 


Figure  G-12. 

Satisfaction  with  Supplies 


Level  of  satisfaction 


Figure  G-13. 

Satisfaction  with  Telephone/Fax 
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Figure  G>14. 

Satisfaction  with  Personal  Computers 


Figure  G-15. 

Satisfaction  with  Clerical  Support 
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Q-7  Are  you  typically  reimbursed  for  per  diem,  travel,  or  lodging  expenses? 


Reimbursed  for  expenses 


Figure  G-16. 

Reimbursement  for  Expenses 


Q-8  How  satisfied  are  you  with  NASA's  expense  reimbursement? 


Figure  G-17. 

Satisfaction  with  Per  Diem 
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Level  of  satisfaction 

Figure  G-18. 

Satisfaction  with  Travel  Reimbursement 


Level  of  satisfaction 

Figure  G-19. 

Satisfaction  with  Lodging  Reimbursement 
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Q-9  Should  NASA  consider  using  information  technology  for  its  science  grant 
process? 


Figure  G-20. 

Use  of  Information  Technology  for  Proposal  Preparation 


Figure  G-21 . 

Use  oflnformation  Technology  for  Proposal  Submission 
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Figure  G-22. 

Use  of  Information  Technology  for  Proposal  Evaluation 


Use  technology 


Figure  G-23. 

Use  oflnformation  Technology  for  Grant  Award 
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Figure  G-24. 

Use  of  Information  Technology  for  Evaluation  Notification 


General 

Q-10  How  satisfied  are  you  with  the  qualifications  of  fellow  panel  members  or 
the  quality  of  mail-in  reviews? 


Percent 
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LEGEND: 


1  =  Not  Satisfied 

2  =  Slightly  Satisfied 

3  =  Fairly  Satisfied 

4  =  Very  Satisfied 

5  =  Extremely  Satisfied 


Level  of  satisfaction 


Figure  G-25. 

Satisfaction  with  Quality  of  Mail-In  Reviews 
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Level  of  satisfaction 


LEGEND: 

1  =  Not  Satisfied 

2  =  Slightly  Satisfied 

3  =  Fairly  Satisfied 

4  =  Very  Satisfied 

5  =  Extremely  Satisfied 


Figure  G-26. 

Satisfaction  with  Panel  Qualifications 


Q-ll  How  satisfied  are  you  that  the  NASA  grant  process  fairly  evaluates  each 
proposal? 


Level  of  satisfaction 


LEGEND: 

1  =  Not  Satisfied 

2  =  Slightly  Satisfied 

3  =  Fairly  Satisfied 

4  =  Very  Satisfied 

5  =  Extremely  Satisfied 


Figure  G-27. 

Satisfaction  with  Evaluation  Fairness 
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Q-12  How  satisfied  are  you  with  NASA's  procedures  for  avoiding  conflicts  of 
interest  or  protecting  proprietary  ideas? 


Level  of  satisfaction 


LEGEND: 

1  =  Not  Satisfied 

2  =  Slightly  Satisfied 

3  =  Fairly  Satisfied 

4  =  Very  Satisfied 

5  =  Extremely  Satisfied 


Figure  G-28. 

Satisfaction  with  Avoidance  of  Conflict  of  Interest 


Level  of  satisfaction 


LEGEND: 

1  =  Not  Satisfied 

2  =  Slightly  Satisfied 

3  =  Fairly  Satisfied 

4  =  Very  Satisfied 

5  =  Extremely  Satisfied 


Figure  G-29. 

Satisfaction  with  Protecting  Proprietary  Ideas 
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Q-13  Are  you  satisfied  that  NASA's  science  grant  process  results  in  the  selection 
of  the  best  research  projects? 


LEGEND: 

1  =  Not  Satisfied 

2  =  Slightly  Satisfied 

3  =  Fairly  Satisfied 

4  =  Very  Satisfied 

5  =  Extremely  Satisfied 


Figure  G-30. 

Satisfaction  that  Process  Yields  Best  Research 


Q-14  Selection  of  the  best  research  projects  and  overall  satisfaction  with  NASA's 
science  grant  process  are  outcomes  that  depend  upon  a  number  of  process 
elements.  How  important  to  those  outcomes  are  the  following  NASA 
process  elements? 


Importance  of  process  element 


LEGEND: 

1  =  Not  Important 

2  =  Slightly  Important 

3  =  Fairly  Important 

4  =  Very  Important 

5  =  Extremely  Important 


Figure  G-31 . 

Importance  of  Time/ Workload 
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Importance  of  process  element 


Figure  G-32. 

Importance  of  Evaluation  Guidelines 


Importance  of  process  element 


Figure  G-33. 

Importance  of  Panel  Support 
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Percent 


LEGEND: 

1  =  Not  Important 

2  =  Slightly  Important 

3  =  Fairly  Important 

4  =  Very  Important 

5  =  Extremely  Important 


Importance  of  process  element 


Figure  G-34. 

Importance  of  Panel  Qualifications 


Figure  G-35. 

Importance  of  Quality  of  Mail-In  Reviews 
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Percei 


70j  67.54% 


Importance  of  process  element 


Figure  G-36. 

Importance  of  Review  Fairness 


Figure  G-37. 

Importance  of  Ethical  Issues 


Q-15  How  satisfied  are  you  with  the  overall  NASA  science  grant  process? 


Figure  G-38. 

Satisfaction  with  Overall  Grant  Process 


Background 

Q-16  What  is  your  general  area  of  scientific  concern? 


LEGEND: 

1  =  Solar  System 

Exploration 

2  =  Astrophysics 

3  =  Space  Physics 

4  =  Other 


Scientific  concern 


Figure  G-39. 

Disciplines 
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Table  G-1. 

Other  Disciplines 


Category 

Frequency 

Percentage 

Atmospheric  chemicals 

3 

27.20 

Biochemistry 

1 

9.10 

Geochemistry 

1 

9.10 

Geophysics 

1 

9.10 

Laboratory  astrophysics 

2 

18.20 

Planetary  material 

1 

9.10 

Spectral  lasers 

1 

9.10 

Technology 

1 

9.10 

Total 

11 

100.00 

Table  G-2. 

Sub  disciplines 


Category 

Frequency 

Percentage 

Astronomy 

4 

2.60 

Atmospheres 

2 

1.30 

Atomic  and  molecular 

2 

1.30 

Chemical  physics 

1 

0.60 

Cosmology 

5 

3.30 

CVS,  LMXRBS 

1 

0.60 

Energy 

1 

0.60 

Exobiology 

4 

2.60 

Extragalactic 

8 

5.30 

Galactic  astronomy 

2 

1.30 

Gamma  ray  astronomy 

2 

1.30 

Geochemistry 

2 

1.30 

Geology 

2 

1.30 

Geophysics 

6 

4.00 

Geoscience 

1 

0.60 

Giant  planets 

1 

0.60 

High  energy 

13 

8.50 

Infrared  observation 

2 

1.30 

Instrumentation 

2 

1.30 

Interstellar 

2 

1.30 

Ionosphere 

3 

2.00 
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Table  G-2. 

Subdisciplines  (Continued) 


Category 

Frequency 

Percentage 

IR 

1 

0.60 

ISM/OB  STARS 

1 

0.60 

Isotope 

1 

0.60 

Late-type  stars 

1 

0.60 

Magnetohydrodyna. 

1 

0.60 

Magnetosphere 

8 

5.30 

Mars,  Venus 

1 

0.60 

Molecular  ISM 

1 

0.60 

Multi  wavelength 

1 

0.60 

Night  vision 

1 

0.60 

Observational 

2 

1.30 

Optical/infrared 

3 

2.00 

Origin  of  solar 

1 

0.60 

Outer  planets 

1 

0.60 

PGG 

1 

0.60 

Planetary  astronomy 

5 

3.30 

Planetary  atmosphere 

5 

3.30 

Planetary  dynamics 

1 

0.60 

Planetary  geology 

2 

1.30 

Planetary  material 

3 

2.00 

Planetary  science 

2 

1.30 

Quasar 

1 

0.60 

Radio  astronomy 

2 

1.30 

Radio  Galactic 

1 

0.60 

Radio  science 

1 

0.60 

Remote  sensing 

3 

2.00 

Small  bodies 

3 

2.00 

Small  bodies 

1 

0.60 

Solar  and  interplanetary 

3 

2.00 

Solar  physics 

11 

7.10 

Solar  system  dynamics 

3 

2.00 

Star  formation 

2 

1.30 

Stars 

2 

1.30 

Stellar  atmosphere 

1 

0.60 

Theoretical  astronomy 

1 

0.60 

Theory  and  high  energy 

1 

0.60 

Theory,  extragalactic 

1 

0.60 

Upper  atmosphere 

1 

0.60 
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Table  G-2. 

Sub  disciplines  (Continued) 


Category 

Frequency 

Percentage 

UV  and  X-ray 

1 

0.60 

X-ray  astronomy 

4 

2.60 

X-ray  and  radio 

1 

0.60 

X-ray  astrophysic 

1 

0.60 

X-ray  observation 

1 

0.60 

Total 

155 

100.00 

Q-17  How  long  have  you  worked  in  this  area? 


<7  7-12  13-18  19-24  >24 

Working  years 


Figure  G-40. 

Years  in  Scientific  Concern 


G-25 


Q-18  How  many  times  have  you  ever  participated  on  NASA  review  panels? 


Times  on  review  panels 


Figure  G-41. 

Panel  Participation 


G-26 


Appendix  H 


Survey  Comments: 
Peer  Review  Panelists 


Survey  Comments:  Peer  Review  Panelists 


In  this  appendix,  we  quote  the  comments  made  by  the  peer  review  panelists 
on  the  science  grant  survey.  (The  comments  have  been  edited  slightly  for  clarity, 
brevity;  and  grammar.)  We  have  organized  the  comments  according  to  the  sub¬ 
sections  in  the  survey: 


Subsection 
Time  or  Workload 

Guidelines  . 

Support  . 

General  . 


H-3 


.  H-5 
.  H-9 
H-1 1 


Time  or  Workload 

♦  The  time  allocated  both  for  the  pre-meeting  review  and  the  review  process 
itself  needs  to  be  increased.  The  reviewers  are  extremely  rushed  during  all 
aspects  of  the  process. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Not  enough  time  is  allotted  for  panel  discussion  of  the  proposals. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Speed  up  turnaround  time  from  submission  to  notification  of  outcome. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  More  lead-time  [needed]  for  proposal  reviews  by  panel  members. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  There  should  be  more  time  at  panel  meetings  devoted  to  plenary  sessions, 
less  time  devoted  to  writing  justifications.  At  plenary  sessions,  every  panel 
member  should  have  read  the  abstract  of  every  proposal.  Use  established 
techniques  for  reaching  consensus  within  a  group. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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♦  During  the  peer  review  process  there  should  be  sufficient  time  allocated  so 
that  the  panel  members  do  not  work  under  time  pressure. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  On  some  reviews,  estimation  of  available  observing  time  is  inaccurate,  lead¬ 
ing  to  major  revisions  of  the  panel  results  by  the  chairs  after  the  meeting; 
this  is  unacceptable. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Panel  workloads  are  too  high.  Things  happen  much  too  quickly  and  one  in¬ 
dividual's  judgment  (or  ax  to  grind)  can  sway  the  rest  of  the  group  under 
the  pressure  of  "the  need  to  move  on."  Yet,  I  understand  that  (1)  resources 
are  limited,  (2)  individual  reviewers  and  panel  members  have  only  so  much 
time,  and  (3)  the  number  of  proposals  keeps  increasing  every  year. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  main  problem  is  that  there  are  too  many  proposals  per  panel.  In  my  last 
panel,  we  were  promised  a  maximum  of  20  proposals  for  4  panelists.  We  ac¬ 
tually  were  given  over  30!  This  meant  having  to  be  [the]  lead  or  secondary 
reviewer  on  more  than  15  proposals!  That  is  just  too  muchl  Suggest  maxi¬ 
mum  [of]  5  proposals  per  panelist. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  number  of  proposals  to  be  reviewed  [has]  increased.  It  was  already  too 
high  for  the  size  of  the  panel  and  the  panelist  would  now  have  to  write  the 
final  summary  reviews  (too  much  work).  Also,  the  honorarium  was 
dropped.  The  bottom  line  is  that  NASA  greatly  increased  the  workload  for 
the  panelist  and  cut  the  pay  simultaneously. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  most  significant  improvement  would  be  to  reduce  the  number  of  proposals 
per  panel.  On  the  panels  I've  served,  we  would  typically  have  only  10  min¬ 
utes  per  proposal  —  this  is  insufficient  time  for  careful  consideration  and 
unfair  to  proposers  who  may  have  invested  a  great  deal  of  effort  in  the  pro¬ 
posal.  Electronic  participation  in  the  review  would  also  be  quite  helpful.  It 
may  be  impractical  to  conduct  the  entire  review  via  telecom,  but  e-mail  dia¬ 
log  on  each  proposal  for,  say,  2  weeks  prior  to  the  panel  session  would  help 
ensure  that  all  proposals  get  careful  consideration. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  It's  a  catch-22  situation:  The  load  is  too  heavy,  the  time  too  short.  But  add¬ 
ing  more  panels  impacts  uniformity  and  overall  balance,  while  measuring 
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the  length  of  reviews  gets  into  the  area  of  diminishing  returns!  One  can  be 
effective  on  this  only  for  so  long. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Get  the  proposals  to  the  reviewers  sooner,  and  give  the  panel  more  time  to 
reconsider  early  gradings. 


Guidelines 

♦  "Technical  Review"  data  should  be  fed  back  verbatim  to  the  proposal  writ¬ 
ers  (technical  reviewers  may  not  be  correct,  and  sometimes  overstep  the 
boundaries  of  technical  matters). 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Proposal  rating  could  be  improved  by  asking  for  ratings  on  each  criterion 
and  NASA  using  those  to  arrive  at  an  overall  rating. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  feel  a  fair  amount  of  time  is  actually  more  or  less  wasted  going  through  a 
formal  review  of  those  proposals  that  are  obviously  not  going  to  be  ac¬ 
cepted.  I  know  it  would  be  difficult  to  formalize  such  an  approach,  (make  it 
legal,  I  suppose),  but  some  two-stage  process  would  probably  be  much  more 
efficient.  Stage  1,  by  e-mail,  phone,  teleconference,  or  whatever:  weed  out 
the  proposals  that  will  not  be  serious  competitors.  Stage  2,  convene  a 
(smaller)  panel  to  deal  with  the  better  half  of  the  proposals.  It's  amazing 
how  much  agreement  there  usually  is.  [This  two-stage  procedure]  would 
probably  save  30%  of  the  actual  time  spent  on  reviews. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  There  are  too  many  proposals  and  too  little  time  to  evaluate  them.  Institute 
a  2  step  process:  1)  Mail-in  reviews  are  used  to  weed  but  proposals  with  less 
than  very  good  or  excellent  ratings.  2)  The  very  good  and  excellent  propos¬ 
als  are  evaluated  by  the  review  panels.  This  process  will  cut  the  time 
needed  for  the  review  panels  to  consider  inferior  proposals  and  will  allow 
more  consideration  of  the  best  proposals. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  very  much  appreciated  a  new  approach  which  placed  strong  reliance  on 
expert  mail-in  reviews,  leaving  the  main  function  of  the  panel  to  adjudicate 
these  reviews. 
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♦  [Use]  more  subdiscipline  panels  as  opposed  to  "catch-all"  panels. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Feedback  from  program  to  proposers  is  still  sluggish  and  limited.  The  way 
costs  are  evaluated  is  awful.  A  good  project  with  an  inflated  budget  is  still  a 
good  project.  It  should  not  get  a  lower  rating,  just  less  [funding].  Panels 
should  be  allowed  to  evaluate  costs  more  objectively.  Instructions,  should 
be  rewritten  in  a  more  understandable  way.  I  never  understood  what  the 
evaluation  element  "relevance  to  NASA  programs"  meant.  Most  reviewers 
are  only  familiar  with  1  or  2  programs  at  best,  not  all  of  NASA's  programs 
and  missions.  This  should  be  changed  to  "relevance  to  this  program." 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  think  more  redundant  reviewing  by  independent  parties  (e.g.,  double¬ 
blind  reviews)  would  be  beneficial,  with  at  least  some  of  the  review  steps  be¬ 
ing  taken  with  no  knowledge  of  the  proposers'  names.  As  an  example:  Use 
mail-in  reviews  (at  least  3  per  proposal),  with  no  principal  investigator  or 
co-investigator  (PI/CI)  names  available  to  the  reviewers;  Do  the  typical 
panel  reviews:  Step  1:  with  no  PI/ Cl  names  available  to  the  reviewers  —  in¬ 
dependent  of  the  mail-in  reviews.  Step  2:  Combine  grades;  discuss  grade 
inconsistencies  —  all  still  without  knowledge  of  PI/CI  names.  Step  3:  If  im¬ 
portant,  consider  the  names  of  PI/CI.  There  is  too  much  of  the  "good  old 
boy"  network  operating  in  the  Astronomy  review  process. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  In  the  peer  review  process,  eliminate  costs  as  a  consideration.  Panel  mem¬ 
bers  are  not  always  knowledgeable  about  geographical  and  institutional  cost 
differences.  These  matters  should  be  evaluated  by  program  managers. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Reviews  should  not  comment  on  unfamiliar  areas.  Outdated  methods  are 
often  still  supported  although  it  has  been  long  shown  that  they  don't  work. 
There  is  often  prejudice  against  new  methods  even  if  they  have  shown 
promise.  Young  investigators  should  be  given  a  fair  chance  but  still  have  to 
demonstrated  expertise,  ability  to  conduct /organize  work,  attain  relevant 
new  results. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  process  is  pretty  well  refined  and  reasonably  apolitical  and  objective. 
Panel  reviews  would  be  more  efficient  if  the  step-by-step  process  were  laid 
out  explicitly  before  the  meeting. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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♦  Rephrase  rating  categories  so  that  they  make  sense  to  someone  familiar  with 
everyday  English. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  My  biggest  concern  about  the  NASA  proposal  review  process  is  that  too 
many  people  are  writing  too  many  proposals,  and  too  many  other  people 
are  reviewing  them.  In  many  cases,  various  proposals,  often  overlapping, 
are  submitted  to  any  NASA  office  that  will  accept  them.  And  there  is  no 
shortage  of  offices  willing  to  accept  them.  I  think  that  the  proposal  solicita¬ 
tion  procedures  should  be  re-evaluated.  I  suggest  standard  dates  for  sub¬ 
missions  of  proposals,  and  that  key  personnel  should  be  tracked 
nation-wide  for  duplication  of  effort  and  over-commitment  of  time.  I  be¬ 
lieve  that  the  pressure  upon  researchers  to  raise  money  to  support  them¬ 
selves  and  their  activities  leads  to  their  spending  too  much  creative  time  [on 
proposals].  Further,  with  so  many  proposals  to  review,  the  review  process 
itself  is  overloaded,  and  this  detracts  from  the  quality  of  the  review  work  it¬ 
self.  Everything  would  be  better  if  NASA  and  like  agencies,  including  NSF, 
DoD,  NOAA,  DOE,  etc.,  coordinated  their  efforts.  It  would  not  be  a  hard¬ 
ship  if  people  could  only  write  a  finite  number  of  proposals,  with  enough 
funds  requested  to  cover  the  true  cost  of  the  research.  In  fact,  it  would  be  a 
blessing. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  If  NASA  were  to  suggest  that  some  areas  were  currently  more  important  to 
their  programs  than  others  (more  likely  to  be  funded),  this  might  reduce 
proposal  writing  and  time  wasted.  NASA  should  be  "up-front"  with  its 
agenda.  I  feel  that  with  decreasing  research  funds,  there  will  be  emphasis 
placed  on  some  fields  while  others  are  de-emphasized  or  dropped.  Such  de¬ 
cisions  should  be  made  known. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  In  my  experience  there  is  a  tension  between  in-house  NASA  proposals  ver¬ 
sus  outside,  university-based  groups.  Typically  the  panel  is  told  to  rate  pro¬ 
posals  by  scientific  merit  only.  Often,  however,  the  very  best  proposals  are 
submitted  by  a  university-based  group,  which  may  be  eligible  for  NSF  fund¬ 
ing.  If  the  panel  selects  this  proposal,  it  may  be  cutting  off  funding  for  an  in- 
house  NASA  group  which  apparently  is  not  eligible  for  NSF  funding.  I  hesi¬ 
tate  to  suggest  that  the  in-house  proposals  should  be  reviewed  separately, 
but  it  appears  that  some  mechanism  should  be  found  to  keep  these  groups 
going. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  evaluation  process,  particularly  guidelines,  has  improved  greatly  over 
the  years. 
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♦  The  confidentiality  of  panel  members  is  not  strictly  kept  by  all  panel  mem¬ 
bers. 


♦  Panel  meetings  only  make  recommendations  to  NASA.  In  some  cases  [NASA] 
scientists  have  FAR  less  expertise  than  the  panel,  and  their  decisions  reflect 
politics  rather  than  the  best  science.  I  would  like  to  see  panel  recommenda¬ 
tions  made  either  binding,  or  at  least  that  the  NASA  selecting  official  explain 
in  writing  why  he  /she  did  not  follow  panel  recommendations. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  [NASA]  should  allow  for  diversity  in  the  selection  process  —  e.g.,  accept  a 
few  proposals  that  are  more  speculative  in  terms  of  their  science  return.  The 
NASA  process  no  doubt  selects  the  best,  say,  30%  of  the  science  and  "throws 
away"  the  worst  30%;  but  the  40%  in  the  middle  depends  too  much  on  the 
chance  make-up  of  a  given  panel  in  a  given  year.  NASA  should  enforce  its 
criteria,  to  prevent  hedging  on  the  rules.  In  particular,  proposals  which 
overrun  page  limits  typically  are  not  penalized. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  noticed  considerable  variation  in  how  a  proposal  fared  based  on  how  it 
was  prepared.  Many  proposals  did  not  break  up  research  tasks,  for  in¬ 
stance,  whereas  some  had  several  discrete  tasks.  In  our  group,  if  several 
tasks  were  delineated,  we  often  voted  on  the  merits  of  each  individually. 
We  often  recommended  partial  funding  to  support  the  individual  tasks  but 
excluded  others.  In  other  words,  the  panel  rewrote  the  proposal  in  certain 
cases.  Those  proposals  that  contained  only  one  task  [received  a  single] 
pass/ fail.  If  we  are  to  invoke  the  policy  of  evaluating  proposals  by  task,  that 
should  be  spelled  out  in  the  instructions  for  proposal  preparation  in  the 
NRA. 

No  graduate  students  should  be  supported  through  research  proposals.  In¬ 
stead,  establish  a  graduate  student  program  wherein  students  at  universities 
entering  into  their  thesis  years  compete  for  educational  funds  to  complete 
their  thesis  in  planetary  studies  (or  better  yet  to  integrate  planetary  studies 
with  other  disciplines).  This  promotes  excellence  in  the  student  not  the  the¬ 
sis  advisor,  and  gives  NASA  control  over  the  number  of  graduate  students 
required  to  ensure  continuation  of  the  planetary  "brain  trust."  I  would  rec¬ 
ommend  that,  for  the  next  few  years,  very  few,  if  any,  graduate  students  be 
supported  by  NASA.  We  have  a  serious  overpopulation  problem.  Consider 
imposing  constraints  on  overhead  rates. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  There  are  two  areas  that  would  improve  the  review  process.  First,  proposal 
grades  have  been  inflated  so  we  are  deciding  the  fates  of  grants  based  on 
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tiny  grade  differentials  where  one  bad  or  inaccurate  review  can  cause  a  pro¬ 
posal  to  fail.  Change  the  grades  to  a  0  - 10  numeric  system  on  each  evalua¬ 
tion  criterion.  Second,  we  come  to  a  meeting  completely  "cold"  on  our 
proposals.  Increase  the  information  flow  before  the  meeting.  Get  primary 
secondary  reviewers  to  submit  their  reviews  at  the  same  time  as  the  external 
[reviews.]  Send  the  complete  set  of  results  to  the  primary  and  secondary 
panel  reviewer,  so  they  can  check  the  "controversial  ones."  They  can  then 
come  to  the  meeting  informed  and  even  draft  the  evaluations  where  there  is 
essential  agreement. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  A  summary  of  decision  ratings  at  end  of  panel  would  be  good.  After  win¬ 
ners  and  losers  are  notified  the  results  should  be  made  public. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  There  are  no  uniform  standards  across  programs.  There  should  be. 

Support 

♦  Currently,  at  [a  federal  laboratory],  we  must  take  leave  without  pay  to  at¬ 
tend  panel  reviews.  In  addition,  we  are  not  allowed  to  collect  any  compen¬ 
sation  (via  an  "honorarium")  for  our  time  spent  on  the  panel  review.  This 
causes  a  large  financial  hardship  for  panel  members,  particularly  those  who 
must  hire  a  day-care  center  or  baby-sitter  to  take  care  of  their  kids  while 
they  are  out  of  town.  Because  of  this,  we  simply  can  no  longer  afford  to  at¬ 
tend  panel  reviews. 

In  addition,  we  are  often  required  to  review  large  numbers  of  proposals  (for 
write-in  reviews)  in  addition  to  those  that  we  judge  as  primary  or  secondary 
reviewers.  A  careful  review  of  [so]  many  proposals  often  takes  more  time 
than  our  tasks  are  funded  for  during  the  fiscal  year.  Special  accounts  are 
needed  to  pay  for  this  review  process.  We  can  no  longer  absorb  this  in  our 
ever-shrinking  research  and  analysis  activities. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  [Need]  more  flexibility  in  travel  arrangements. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  NASA  should  pay  a  consulting  fee,  in  addition  to  per  diem,  travel,  etc.,  to  its 
panel  members. 
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♦  The  most  important  consideration,  in  my  view,  is  to  get  the  best  people  pos¬ 
sible  [as]  reviewers.  Even  if  there  are  lots  of  conflicts,  I  would  still  recom¬ 
mend  getting  the  best  reviewers.  How  about  modest  honoria,  say  $1,000? 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  In  the  panel  review,  reimbursements  should  be  handled  separately  from  ac¬ 
tual  compensation  (payments)  to  simplify  tax  preparation. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  My  main  complaint  about  participating  on  panels  is  that  I  am  spending  a 
substantial  amount  of  time  when  I  am  being  paid  by  someone  else  to  do 
something  different.  I  don't  think  [charging  to  another  grant]  should  sup¬ 
port  NASA  activities.  NASA  should  pay  my  salary  for  my  time  on  [NASA 
work]. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Try  video  conferencing  to  reduce  travel. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  system  for  reimbursing  travel  makes  it  difficult  for  civil  service  scientist 
to  participate,  because  of  artificial  restrictions  on  Center  travel  budgets.  This 
can  be  relieved  if  HQ  were  to  retain  travel  dollars  for  this  purpose. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  It  seems  to  me  that  there  is  some  waste  in  the  arrangement  of  meeting  ex¬ 
penses.  For  example,  whenever  I  come  to  serve  on  a  panel  I  can  purchase 
my  ticket  50%  cheaper  than  what  is  offered  to  me  by  the  official  travel  office. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  NASA  should  provide  monetary  reimbursement  for  extra  review  work.  Kitt 
Peak  National  Observatory  pays  its  committee  members  for  this;  NASA 
should  be  able  to  do  this  also. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Reviews  run  by  Goddard  x-ray  community  are  excellent.  Superb  arrange¬ 
ments,  panels,  tech  support,  etc. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  believe  the  process  works  well  and  would  not  successfully  be  replaced  by  a 
mail-in  process  only.  To  keep  cost  down,  I  suggest  that  it  may  be  advanta¬ 
geous  to  organize  tire  reviews  over  weekends,  which  would  reduce  airfares 
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by  two-thirds.  Mixed  panels  are  good  from  the  point  of  view  of  fairness,  but 
they  sometimes  result  in  too  little  expertise  in  a  given  panel  or  for  a  given 
proposal. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Travel  reimbursement  is  too  slow.  The  hotel  bill  should  be  directly  billed  to 
NASA. 


General 

♦  I  have  an  impression  there  is  a  great  deal  of  allocation/ denial  of  funds  for 
"programmatic"  reasons  which  could  include  strong  personal  biases  of  the 
HQ  staff.  This  point  should  be  open  to  the  panel's  scrutiny.  The  process  is 
too  expensive;  when  scientists  are  struggling  to  get  a  month's  salary  or 
funds  to  support  a  student,  it  is  preposterous  to  be  spending  the  equivalent 
of  several  research  grants  on  panel  meetings. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  problem  with  the  NASA  grant  program  has  nothing  to  do  with  proce¬ 
dure!!!  There  is  simply  too  little  money  to  fund  the  many  excellent  propos¬ 
als.  Therefore,  good  proposals  are  unfunded,  and  inevitably  people  are 
unhappy.  Given  the  current  level  of  funding,  no  procedure  is  satisfactory  — 
all  one  can  hope  for  is  to  avoid  blatant  unfairness,  which  I  think  is  done. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  quality  and  experience  of  the  panel  is  really  quite  important.  In  general, 
NASA  does  a  careful  job  in  selections,  but  was  well  below  the  mark  (as  well 
as  appearing  very  disorganized),  during  the  ATP  peer  review  in  the  summer 
of  1994.  I  hope  and  believe  that  this  was  anomalous. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  My  experience  on  several  occasions  has  been  that  the  review  panel  meets  in 
Washington  and  does,  for  the  most  part,  a  very  careful  and  competent  job  of 
weighting  the  various  issues,  criteria,  and  arriving  at  a  ranking  of  the  pro¬ 
posals  that  is  a  fair  and  accurate  assessment  of  their  scientific  merit.  We 
leave  and  I  later  meet  a  colleague  at  other  meetings  to  discover  to  my  dis¬ 
may  that  top-ranked  proposals  were  not  funded  while  others  further  down 
the  list  were  funded!  I  can  only  conclude  that  the  Administrators  make 
awards  based  on  considerations  other  than  the  findings  of  the  panel  review. 
This  leaves  many  of  us  frustrated  with  the  system.  I  hope  this  survey  may 
prompt  some  reform,  as  the  top-quality  proposals  are  not  being  funded. 
Thank  you. 
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♦  One  basic  problem  is  structural:  a  differentiation  between  "new  starts"  and 
"continuing"  investigators  who  are  proposing  a  new  (3  year,  normally)  in¬ 
crement  of  NASA  funding.  The  external  reviewers  don't  know  of  this  dis¬ 
tinction;  the  proposers  (many  of  them)  also  do  not  know,  but  it  is  on  this 
issue  that  NASA's  programmatic  stamp  often  overrides  scientific  merit. 
NASA  should  instead  adopt  a  model  more  akin  to  NSF's:  Banish  "no  new 
starts"  from  program  manager  vocabularies.  Stop  protecting  the  "estab¬ 
lished"  investigators  and  their  agendas.  This  is  harmful  to  the  science! 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Every  effort  must  be  taken  to  ensure  ethical  behavior  of  reviewers.  No  panel 
is  ever  free  from  all  conflicts  of  interest  —  the  fields  are  too  small.  Panel 
chairs  and  NASA  representatives  must  always  be  aware  of  potential  con¬ 
flicts. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Conflict  of  interest  has  become  extreme  both  on  the  part  of  the  "outside"  re¬ 
views  and  the  NASA  representatives.  The  conflict  [between]  employees  of 
NASA  centers  is  particularly  severe.  As  funding  shrinks  the  committees 
stress  "reliable"  projects,  "killing"  innovation.  Also,  there  are  inadequate 
funds  being  distributed  to  many  grants. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  In  some  cases,  the  proposal  rating  was  only  determined  (voted)  by  the  pri¬ 
mary  and  secondary  reviewers  rather  than  the  entire  panel.  To  ensure  fair¬ 
ness,  the  rating  of  every  proposal  should  be  voted  on  by  all  the  panel 
members,  who  also  [should]  read  the  proposal  (but  may  not  as  carefully  as 
the  primary  and  secondary  reviewers).  Such  instructions  should  be  given  to 
the  chairs  before  the  review  starts.  Scientific  merit  should  always  be  the 
highest  priority  in  the  review  evaluation  criteria.  The  current  situation  that 
junior  researchers  [have  difficulty  in  obtaining]  long-term  grants,  is  partially 
due  to  the  fact  that  panel  members  are  not  real  experts  in  the  field  and  they 
are  more  likely  to  [select]  senior  people.  So,  they  give  awards  to  the  "big 
names."  I  suggest  that  the  panel  members  consist  of  both  senior  and  junior 
scientists.  A  junior  expert  is  more  likely  to  make  a  fair  judgment  than  a  sen¬ 
ior  [expert].  A  survey  should  be  conducted  after  every  review  process,  i.e.,  a 
survey  sheet  is  given  to  every  panel  member  at  the  end  of  the  meeting.  This 
may  help  to  reveal  and  solve  potential  problems  on  a  timely  basis. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  NASA  relies  too  much  on  the  panel  reviews,  with  very  little  emphasis  on  the 
written  review  process.  I  think  NSF  has  a  much  better  grant  review  process, 
because  they  rely  much  more  on  written  reviews.  NASA  panel  reviews  do 
not  allow  one  to  obtain  the  best  possible  expertise  for  each  proposal,  due  to 
the  limited  number  of  panel  members.  Often,  there  are  2  or  fewer  written 
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reviews  (as  compared  to  6  for  NSF)  for  each  proposal  in  a  NASA  review  (of¬ 
ten  because  the  requests  for  written  reviews  are  treated  as  an  afterthought 
rather  than  of  primary  importance  in  the  NASA  review  process).  Panel  re¬ 
views  lead  to  one  reviewer  (the  primary)  having  more  than  90%  weight  in 
the  treatment  of  a  proposal. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Peer  review  panels  do  not  work!  The  old  system  (like  that  employed  at 
NSF),  with  mail-in  reviews  and  management  decisions  made  at  the  program 
director  level  is  far  superior  to  the  panel  system.  By  this,  I  refer  generally'  to 
the  Rand  A  programs.  Missions  selection  works  a  bit  better  under  the  panel 
review  system.  My  experience  is  that  the  best  few  proposals  rise  to  the  top 
and  get  funded.  However,  in  a  competitive  world,  any  objective  sorting  of 
most  of  the  proposals  is  not  accomplished.  I  would  guess  that  the  top  half 
of  the  proposals  which  are  funded  deserves  to  be  funded,  but  the  bottom 
half  is  easily  interchangeable  with  at  least  an  equal  number  of  the  best  of  the 
unfunded.  This  large  uncertainty  is  decided  finally  by  the  particular  preju¬ 
dices,  knowledge,  and  sub-disciplines  of  the  particular  panel  assembled. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  think  the  process  of  grant  selection  is  as  good  as  it  can  realistically  be  but 
of  course  [is]  not  perfect.  Of  more  concern  is  the  monetary  amount  associ¬ 
ated  with  each  awarded  grant.  On  panels  I've  served,  the  typical  grant  is  on 
the  order  of  $10  -  20K  which  is  generally  not  enough  to  ensure  that  the  data 
analysis  effort  is  adequately  supported. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  NSF  review  process  appears  to  be  more  efficient.  NASA  should  investi¬ 
gate  ways  to  reduce  the  percentage  of  available  funds  required  to  manage 
programs  and  evaluate  proposals. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  believe  that  the  NASA  program  office  should  try  harder  to  stay  on  course 
and  not  be  jostled  about  by  every  wind  of  fashion  that  comes  along.  It 
should  also  try  to  be  more  consistent  in  its  announcement  of  opportunities. 
Several  times  I  have  spent  considerable  effort  in  writing  proposals,  follow¬ 
ing  a  NASA  announcement,  only  to  be  told  that  there  was  no  money  for  this 
program  and  my  proposal  simply  wouldn't  be  reviewed.  It  makes  NASA 
headquarters  appear,  rightly  or  wrongly,  as  if  they  are  extremely  uncertain 
what  course  they  are  following. 

I  believe  there  could  be  a  stronger  effort  towards  communication  with  and 
oversight  of  planetary  science  community  once  the  funds  have  been 
awarded.  There  are  some  groups  that  would  rather  work  12  hours  a  day  to 
make  sure  that  they  comply  with  the  commitment  to  the  proposal.  On  the 
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other  hand,  there  are  others  who  look  upon  successful  proposals  as  some 
kind  of  a  game  with  only  a  minimal  effort  required  to  carry  out  the  requisite 
work.  At  present  I  get  the  feeling  that  NASA  headquarters  isn't  aware  of 
this  or  perhaps  doesn't  care. 

Parenthetically  this  problem  was  much  better  addressed  under  the  older 
system,  where  the  program  administrator  reviewed  the  proposals  himself. 
He  was  thus  entirely  responsible  for  the  choices  he  made.  He  made  trips  to 
visit  various  groups,  discussed  their  needs  and  saw  how  they  were  doing. 
He  grudgingly  even  admitted  some  praise  for  work  done,  while  the  present 
outlook  seems  to  concentrate  mainly  on  cutting  the  budget  and  making  in¬ 
vestigators  continue  the  same  level  of  work  for  less  funding.  The  older  sys¬ 
tem  worked  well  because  the  administrator  of  the  program  was  extremely 
dedicated  and  capable.  It  would  not  work  well  if  there  was  a  less  capable 
person  at  the  helm,  and,  because  of  this  possibility,  I  do  not  want  to  suggest 
abolishing  the  present  scientific  advisory  committees  which  are  extremely 
important. 

If  there  is  more  of  an  oversight  function  and  more  involvement  by  the  pro¬ 
gram  administrator,  the  actual  work  that  is  done  becomes  more  important. 
Thus,  the  proposals  themselves  could  be  made  shorter,  cutting  their  present 
size  roughly  in  half,  the  review  committee  could  also  be  cut  roughly  in  half 
and  more  emphasis  could  be  given  to  overall  performance  of  the  proposing 
individuals  and  groups. 

Large  spacecraft  proposals,  such  as  the  Discovery  program,  should  be  han¬ 
dled  more  elaborately.  I  have  seen  a  number  of  occasions  where  [such]  pro¬ 
posals  were  given  a  low  rating  because  of  some  misunderstanding  by  the 
committee  of  some  technical  details,  which  were  actually  correct.  It  is  miy 
suggestion  that  the  principal  investigator  for  these  proposals  be  invited  to 
give  a  short  20  minute  presentation  about  his  project  and  his  philosophy  in 
front  of  the  panel.  This  is  followed  by  a  30  minute  question  and  answer  ses¬ 
sion.  The  panel  then  gets  a  chance  to  ask  questions  about  specific  items  of 
the  proposal  which  had  been  flagged,  and  the  [proposal  writers]  get  a 
chance  to  defend  their  approach  and  display  the  science  and  its  technical 
feasibility.  While  it  may  seem  that  this  extra  step  is  an  added  burden,  I  feel 
it  would  make  the  selection  procedure  fairer  and  result  in  better  projects  for 
NASA. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Rejection  is  often  based  on  technical  aspects  of  proposal  preparation  (e.g., 
did  not  address  how  project  fit  into  NASA  goals;  no  detailed  work  plan). 
Such  proposals  are  dropped  immediately  by  panels,  irrespective  of  merit. 
Too  much  emphasis  on  proposal  formalism! 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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♦  I  have  been  impressed  with  the  quality  of  the  evaluation  panels  I  have  par¬ 
ticipated  in  for  the  astrophysics  branch.  The  panel  results  have  always  been 
fairly  based  on  scientific  merit  and  utility.  I  can't  say  how  closely  the  final 
selection  and  funding  had  adhered  to  the  panel  recommendation,  but  I  have 
heard  little  complaint  from  the  astrophysics  community.  However,  I  have 
heard  complaints  from  planetary  colleges  concerning  the  fairness  of  their  re¬ 
views. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  In  a  few  cases,  principal  investigators  who  have  not  been  productive  are 
automatically  continued  on  the  basis  of  outdated  reputations!  This  should 
be  discouraged.  Residence  time  required  in  Houston  is  too  long:  more  elec¬ 
tronic  communications  among  panels,  in  advance,  could  shorten  process  in 
Houston  to  3  days;  the  current  8  days  is  unacceptable  for  people  who  have 
real  lives  and  other  responsibilities. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Modified,  or  not,  conflict  of  interest  regulations  are  imperative:  the  current 
legislative  mumbo-jumbo  prevents  many  reviewers  with  no  real  conflict 
from  participating  and,  at  the  same  time,  allows  the  participation  of  review¬ 
ers  with  serious  conflicts  of  interest.  Electronic  submission  and  circulation 
(to  reviewers)  of  proposals  should  be  adopted  immediately.  Electronic  peer 
review  should  be  investigated  [to  reduce]  travel-time  and  costs. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  most  serious  flaw  in  the  evaluation  program  may  be  in  assessing  the 
"quality  vs.  quantity"  issue.  I  have  always  gone  the  route  of  producing  a 
few  high-quality  papers  each  year  instead  of  an  abundance  of  mediocre  pa¬ 
pers.  But,  reviewers  often  only  have  time  to  look  over  the  bulk  of  publica¬ 
tions  a  proposer  has  published  rather  than  the  quality  of  those  works. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  process  is  incredibly  laborious  and  —  therefore  —  costly.  A  very  large 
[proportion]  of  the  money  available  for  research  is  spent  in  the  process  of 
deciding  who  will  get  grants.  I  implore  NASA  to  investigate  ways  to  fairly 
[select]  proposals  more  cheaply. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  At  present  there  is  too  much  good  science  chasing  too  few  dollars.  This  in¬ 
troduces  a  random  component  in  the  funding  process,  and  too  many  good 
projects  lose  out.  The  response  is  to  keep  trying,  and  as  a  result,  we  are 
drowning  in  proposals  —  both  writing  them  and  reading  them.  Drastic  re¬ 
visions  in  the  process  are  needed.  Maybe  the  time  has  come  for  term  limita¬ 
tions.  A  strict  limit  on  the  number  of  proposals  that  any  investigator  may 
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submit  to  an  government  agency  in  a  given  year;  and  a  strict  limit  on  the 
number  of  continuous  years  that  any  investigator  may  be  funded  in  a  given 
program. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  review  process  has  become  so  competitive  that  vast  resources  are  spent 
in  both  preparing  and  reviewing  proposals,  75%  of  which  are  not  funded. 
Thus  we  have  reached  a  point  of  diminishing  returns.  Furthermore,  the 
process  favors  "safe  research,"  projects  where  the  outcome  is  fairly  predict¬ 
able  and  well  defined.  Innovative  research  with  new  ideas  becomes  too 
risky  to  fund. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  think  the  process  works  as  well  as  can  be  expected.  I  believe  it  is  fair  and 
the  reviewers  [are]  largely  free  of  personal  bias  in  their  assessments.  Some 
clarification  of  the  "conflict  of  interest"  rules  is  needed.  It  seems  to  go  to 
awkward  lengths  at  times  (indeed  to  the  extent  of  eliminating  the  majority 
of  the  panel)  with  little  justification.  I  have  found  the  mail-in  reviews  to  be 
very  [uneven]  in  quality.  I  believe  the  process  could  be  improved  but  Tm 
not  sure  how  (shorter  guidelines?).  The  panel  reviews  are  the  real  determi¬ 
nant  and  the  primary  reviewer  has  the  main  voice  in  the  outcome. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  process  is  perfect  if  there  is  money.  For  the  last  five  years,  it  has  been  a 
waste  of  time  and  money.  Stop  it  until  you  can  [improve  it].  Until  then  use 
outside  reviews  and  contract  monitors.  I  trust  the  contract  monitors. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Our  panel  was  very  disturbed  at  the  small  amount  of  funds  to  be  distributed 
to  a  very  large  number  of  outstanding  proposals.  It  didn't  seem  worth  the 
travel  money  and  time  to  assemble  all  of  us  to  debate  giving  out  so  little 
money. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  In  continuing  programs,  it  would  be  useful  to  have:  1)  A  list  of  previously 
funded  proposals  sorted  by  PI  and  also  by  subject.  2)  [A  list  of]  Pis  who 
have  previously  [received]  awards,  a  list  of  these  awards  and  relevant  pro¬ 
gress  reports. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Chief  reviewers  should  be  given  names  and  e-mail  addresses  of  mail-in  re¬ 
viewers  for  their  proposals,  and  it  should  be  their  responsibility  to  call  and 
make  sure  mail-in  reviews  are  in  on  time.  The  last  panel  I  served  on  had 
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several  proposals  with  no  mail-in  reviews.  This  puts  the  panel  in  a  position 
of  having  to  review  proposals  for  which  there  may  be  no  expertise  on  the 
panel,  and  it  is  very  unfair  to  the  potential  [principal  investigator].  A  small 
pool  of  money  should  be  made  available  to  new  Pis,  or  at  least  consideration 
should  be  given  so  that  all  money  does  not  go  to  ongoing  projects  all  the 
time,  even  if  these  are  highly  rated  by  the  existing  community  (who  have  an 
interest  in  the  status  quo).  It  seems  difficult  to  protect  proprietary  ideas  ex¬ 
cept  to  use  sanctions  against  persons  who  abuse  privilege. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  My  main  concern  is  with  fairness  and  preparation  of  the  panel  members. 
Both  times  I  sat  on  panels  there  were  few  if  any  mail-in  reviews.  This  leaves 
too  much  to  the  panel  members.  If  one  member  doesn't  conscientiously  do 
his  or  her  job,  then  proposal  [evaluations]  suffer.  Panel  members  should 
also  be  forewamed/admonished  that  they  must  come  prepared.  I  have 
seen  terribly  under-prepared  panel  members  and  it  severely  compromises 
the  process. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Panels  should  be  divided  more  into  subcategories  like  the  NSF  panel  review, 
so  that  cross-discipline  bias  is  removed. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  More  people  would  be  available  for  panel  service  if  we  didn't  have  to  write 
so  many  separate  proposals.  Maybe  there  should  be  more  tolerance  for 
multiple-task  (not  closely  related)  proposals.  It  would  give  NASA  more 
"bang  for  the  buck." 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Peer  review  panels  do  not  work!  The  old  system  (like  that  employed  at 
NSF),  with  mail  reviews  and  management  decisions  made  at  the  program 
director  level,  is  far  superior  to  the  panel  system.  By  this  I  refer  generally  to 
the  Rand  A  programs.  Mission  selection  works  a  bit  better  under  the  panel 
review  system.  My  experience  is  that  the  best  few  proposals  rise  to  the  top 
and  get  funded.  However,  in  a  competitive  world,  any  objective  sorting  of 
most  of  the  proposals  is  not  accomplished.  I  would  guess  that  the  top  half 
of  the  proposals  which  are  funded  deserve  to  be  funded,  but  the  bottom  half 
are  easily  interchangeable  with  at  least  an  equal  number  of  the  best  of  the 
unfunded.  This  large  uncertainty  is  decided  finally  by  the  particular  preju¬ 
dices,  knowledge,  and  subdisciplines  of  the  particular  panel  assembled. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Funding  Process:  Increase  grant  sizes  to  reduce  overhead  in  writing  and  re¬ 
viewing  process.  Increase  grant  length  for  selected  proposals/areas.  Don't 
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support  just  for  social  reasons;  should  be  at  most  secondary  consideration, 
as  long  as  money  is  extremely  tight. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  most  difficult  challenge  is  controlling  subjective,  personal,  "conflict  of 
interest"  factors  injected  by  a  small  minority  of  panelists.  I  have  the  highest 
respect  for  the  effectiveness  of  the  public  officials  I  have  encountered  who 
are  charged  with  conducting  these  reviews. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  In  multi-discipline,  multi-mission  program  reviews  there  is  often  an  implicit 
"quota"  of  how  [much]  money  will  be  divided  up  according  to  discipline  or 
mission  depending  on  the  composition  and  relative  "weights"  of  the  panels. 
Hence,  it  may  help  the  proposers  to  gain  insights  into  their  chances,  inde¬ 
pendent  of  the  quality  of  the  proposal.  This  is  a  very  important  issue  since 
most  of  the  "unfairness"  complaints  originate  from  such  types  of  reviews. 
The  more  narrowly  focused,  mission-oriented  reviews  are  usually  consid¬ 
ered  fair  by  most  colleagues. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  big  problem  is  that  there  are  now  so  many  deadlines,  proposals  and  re¬ 
views,  that  the  relatively  small  astrophysics  community  is  in  some  danger  of 
becoming  gridlocked.  It  would  be  tragic  for  NASA  if  people  end  up  spend¬ 
ing  more  time  talking  and  debating  about  what  they  are  going  to  do  than  ac¬ 
tually  doing  it!  NASA  has  been  doing  some  good  things  to  check  this 
dangerous  trend:  much  more  streamlined  and  shorter  proposals  and  re¬ 
views  have  been  a  huge  help  without  any  resulting  sacrifice  of  scientific 
quality.  More  might  be  done,  along  the  lines  of  shorter  proposals  and  fewer, 
bigger  reviews  (i.e.,  for  longer  time  periods).  Keep  panel  size  as  small  as  pos¬ 
sible ! 

♦  I  have  mainly  been  involved  in  reviews  which  awarded  amounts  on  the  or¬ 
der  of  a  few  $100K  or  so  and  I  am  generally  satisfied  with  the  process.  My 
main  concern  is  with  larger  awards.  It  seems  to  me  that  the  multi-hundred 
million  dollar  awards  are  not  given  proportionally  more  scrutiny.  I  think  a 
more  open  review  process,  would  be  appropriate. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Concerns  about  conflict  of  interest  from  individuals  from  the  same  institu¬ 
tion  are  given  too  much  weight  —  interfere  with  giving  proposals  a  bal¬ 
anced  review  by  a  full  panel.  In  my  experience,  bias  arises  when  a  panelist 
is  a  collaborator  of  the  proposer  (often  not  at  the  same  institution),  but  only 
very  rarely  do  institutional  loyalties  surface. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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♦  NASA  does  a  very  good,  thorough  job  of  reviewing  proposals  compared, 
say,  to  NSF  which  gives  too  little  time  for  evaluation.  The  main  threats  to 
peer  reviews  are  congressional  interference  and  the  possibility  that  the  re¬ 
views  (and  reviewers  names)  might  become  accessible  to  the  proposers. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  peer  review  process  works  well  when  permitted  to  operate  at  all;  too  of¬ 
ten,  though,  the  process  is  circumvented. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  have  witnessed  some  pretty  sloppy  reviewing  and  unfair,  uniformed  deci¬ 
sions.  In  one  case  a  proposal  of  one  of  my  colleagues  was  rejected  for  cited 
reasons  of  omission  of  [facts]  that  were  precisely  those  addressed  explicitly 
in  the  proposal.  The  proposal  had  obviously  not  been  read  carefully  per¬ 
haps  because  my  colleague  was  a  first  time  proposer  —  not  one  of  the  "old 
boys"!  I  believe  the  panel  does  not  solicit  outside  reviews,  and,  as  a  result, 
each  panel  member  may  have  too  great  a  workload.  The  outside  reviews  are 
vital  for  obtaining  expertise  that  may  be  missing  on  the  panel. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  cost  (in  both  time  and  money)  to  the  space  physics  community  of  the 
entire  grant  process  (preparation  of  proposals  and  their  subsequent  evalua¬ 
tion)  is  unconscionably  high  relative  to  the  amount  of  money  that  is  avail¬ 
able  to  be  awarded.  With  an  acceptance  rate  of  only  12  - 15%,  the  system 
simply  cannot  continue  in  its  present  form. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Considering  the  relatively  small  amount  of  money  available  for  the  science 
grant  program  would  it  not  make  sense  to  issue  fewer  announcements. 
Most  grants  run  for  two  or  three  years  anyway. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  My  [main]  concern  is  with  the  fairness.  The  personal  preferences  and  preju¬ 
dices  of  some  panel  members  can  have  a  strong  effect  on  the  review.  It 
would  help  if  panel  members  were  provided  (beforehand)  not  only  with  the 
proposals,  but  also  with  reports  from  anonymous  mail-in  reviewers. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  one  thing  needed  is  to  have  career  program  managers.  We  have  a  new 
[program  manager]  detailed  every  two  or  three  years  (sometimes  more  of¬ 
ten)  and  this  spoils  the  continuity  of  policy,  program,  and  program  manager 
expertise.  NASA  also  appears  to  be  reaching  "deeper  into  the  barrel"  as 
time  goes  by. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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♦  Too  much  favoritism  towards  established  investigators.  A  PI,  at  an  estab¬ 
lished  center  has  a  tremendous  advantage  over  a  new  researcher  because  of 
the  existing  facilities  and  his/her  "track  record."  Consequently,  even  if  the 
proposed  research  is  brilliant,  the  new  guy  will  have  two  strikes  against  him 
or  her.  But,  in  addition  to  this  natural  advantage,  there  is  the  policy  that 
once  a  PI  is  funded,  he /she  just  has  to  write  renewals,  which  are  not  evalu¬ 
ated  in  the  same  way  as  new  proposals.  This  amounts  to  lifetime  tenure  for 
investigators  once  they  get  their  foot  in  the  door,  leaving  little  money  for 
new  investigators. 

Too  much  control  during  panel  evaluation  by  the  Chief  Reviewer  (CR),  who 
synthesizes  each  project  he/she  is  in  charge  of  and  provides  a  running  [ac¬ 
count].  Other  panel  members  too  often  defer  to  the  opinion  of  the  CR  be¬ 
cause  they  do  not  have  the  time  to  study  the  proposal  themselves. 
Consequently,  the  evaluation  a  proposal  receives  typically  hinges  on  the 
opinion  of  one  individual,  who  can  extract  the  phrase  from  various  reviews 
to  support  his/her  viewpoint.  This  has  to  be  remedied. 

[There  is]  too  much  variation  in  scoring  from  group  to  group.  I  suspect  that 
each  group  has  a  different  view  of  what  constitutes  good  scores  and  that 
probably  is  reflected  in  the  "threshold"  for  acceptance.  For  groups  encom¬ 
passing  many  different  disciplines,  there  is  less  consensus  among  the  panel¬ 
ists,  partly  due  to  lack  of  expertise,  partly  due  to  other  biases  (some 
approaches  were  just  unpopular,  some  proposals  didn't  have  much  of  a 
chance).  I  know  from  the  group  I  sat  on  there  were  some  fields  that  the 
group  seemed  to  be  more  favorably  inclined  towards  than  others.  This  sug¬ 
gests  that  the  fate  of  a  particular  proposal  could  hinge  upon  which  group 
the  proposal  is  evaluated  by.  I  also  know  from  my  experience  as  an  associ¬ 
ate  reviewer  (involvement  with  another  group)  that  groups  have  fundamen¬ 
tally  different  views  on  assigning  numerical  scores  to  evaluations. 

Too  much  emphasis  is  placed  on  the  number  of  publications  authored  by  the 
investigator  rather  than  the  significance  or  quality  of  the  publications.  The 
principal  effect  of  this  is  that  the  community  is  encouraged  to  practice  "safe 
science,"  i.e.,  they  go  for  modest,  sometimes  trivial,  incremental  advances  in 
the  fields  rather  than  the  bold  significant  studies. 

Eliminate  the  bias  toward  established  Pis.  All  proposals  should  be  treated 
identically.  Encourage  projects  of  3  -  5  year  length,  where  appropriate,  but 
when  that  time  is  over,  the  PI  has  to  write  a  full  proposal  and  is  treated  ex¬ 
actly  like  a  new  investigator. 

Do  not  allow  professors  to  "buy  out  of  teaching."  Place  limits  on  the 
amount  of  summer  salary  a  faculty  member  can  propose  for. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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♦  [I]  have  participated  in  some  reviews  where  the  proposal  was  poorly  written 
and  not  responsive.  We  rejected  or  put  the  proposal  last  on  the  priority  list. 
The  proposal  was  chosen  by  NASA  (the  name  was  well  known)  and  in  one 
case  the  person  was  even  made  team  leader!  This  does  not  leave  a  good  im¬ 
pression  with  new,  young  members  of  the  panel.  Its  the  "good  old  boys" 
problem  —  NASA  needs  to  continually  work  on  this;  its  an  "ethical  issue"!!! 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  panel  process  works  well  for  large  proposals,  where  the  number  of  pro¬ 
posals  is  small;  but,  [it]  is  flawed  for  small  proposals,  where  the  number  is 
larger  and  the  diversity  is  far  greater.  Results  are  subject  to  the  whims  of 
opinionated,  primary  reviewer  panel  members,  and  when  there  are  many 
proposals  under  review,  there  is  little  time  or  energy  for  a  critical  panel 
evaluation.  In  such  a  situation,  I  would  prefer  to  see  a  mail-in  review, 
where  there  would  be  reports  from  several  experts  on  each  proposal. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  More  information  on  who  is  currently  supported  to  do  what,  should  be 
available  to  the  panel  to  aid  in  assessing  program  balance.  Also,  reports 
from  previous  reviews  of  similar  proposals  by  the  same  principal  investiga¬ 
tor  should  be  available  to  aid  consistency. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Better  feedback  to  proposers,  especially  unsuccessful  proposers,  is  needed. 
Use  of  a  system  modeled  after  the  Kitt  Peak  National  Observatory  telescope 
committee  would  be  a  good  place  to  start. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  think  NASA  should  more  actively  solicit  and  listen  to  reviews  of  the  man¬ 
agement  plans  for  large,  complex  projects  from  the  peer  reviewers.  Many 
peer  reviewers  have  been  Pis  or  project  scientists  and  have  very  valuable  ex¬ 
perience  to  bring  to  the  review  of  management  plans.  I  was  on  one  review 
committee  where  the  management  volumes  were  shown. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  would  like  to  see  NASA  officially  recognize  both  a  scientific  PI  and  an  ad¬ 
ministrative  PI  on  all  proposals.  This  would  help  the  reviewers  in  their 
evaluation  (since  often  the  actual  PI  does  not  write  the  proposal)  and  it 
would  also  encourage  better  scientific  competition  and  hence  higher  quality 
research.  Today,  many  young,  talented  scientists  face  discouragement 
when  writing  proposals  because  their  institution  allows  them  no  financial 
responsibility,  hence  they  gain  no  true  recognition  from  NASA  throughout 
the  grant  award  process.  If  NASA  began  recognizing  scientific  Pis,  we 
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would  see  our  tax  money  funding  a  higher  overall  level  of  research  in  the 
community. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Have  panel  chairs  meet  just  before  plenary  sessions  to  make  sure  science 
coverage  is  optimal.  Presently,  chairs  tend  to  meet  too  early  to  determine 
how  things  will  "shake  out,"  then  too  late  to  correct  oversights.  I  strongly 
discourage  panel  members  getting  together  with  other  panel  members  to  in¬ 
fluence  outcomes.  I  have  witnessed  this  sort  of  behavior.  Make  technical  re¬ 
viewers  accountable  for  their  assessments  by  citing  technical  reviewers' 
comments  verbatim  in  proposal  reports  (person  can  remain  anonymous). 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  My  primary  concern  with  the  grant  proposal  process  is  with  conflicts  of  in¬ 
terest.  I  felt  that  there  was  some  strong  bias  on  at  least  one  committee  mem¬ 
bers  part  against  proposals  competing  with  those  from  that  member's 
institution,  and  that  this  greatly  affected  the  outcome  of  the  review. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Because  of  tight  funding,  there  is  a  tendency  to  fund  "safe"  projects.  This  re¬ 
sults  in  Pis  proposing  mostly  safe  ideas,  a  dangerous  development  because 
it  leads  to  the  decline  of  truly  new  and  innovative  research. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Make  sure  that  there  is  an  adequate  number  of  external  mail-in  reviews  (ap¬ 
proximately  four),  so  that  no  single  reviewer  carries  too  much  weight.  The 
community  must  continue  to  guard  against  being  too  in-bred.  In  selection 
of  external  reviewers,  a  broad  representation  of  people  with  expertise  in 
various  aspects  of  a  given  proposal  must  be  chosen.  The  importance  of  a 
program  manager  with  a  clear  and  broad  vision  for  the  program  is  essential. 
Fortunately,  we  have  a  good  one! 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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Appendix  I 


Survey  Questions  and  Responses: 

Proposal  Writers 


Survey  Questions  and  Responses: 
Proposal  Writers 


Time 


Q-l  How  satisfied  are  you  with  the  3-month  period  allotted  for  preparing  grant 
proposals? 


LEGEND: 

1  =  Not  Satisfied 

2  =  Slightly  Satisfied 

3  =  Fairly  Satisfied 

4  =  Very  Satisfied 

5  =  Extremely  Satisfied 


Figure  1-1. 

Satisfaction  with  Proposal  Preparation  Time 


1-3 


Q-2  How  satisfied  are  you  with  NASA's  6-month  standard  for  providing  pro¬ 
posal  feedback? 


LEGEND: 


1  =  Not  Satisfied 

2  ss  Slightly  Satisfied 

3  =  Fairly  Satisfied 

4  =  Very  Satisfied 

5  =  Extremely  Satisfied 


Figure  1-2. 

Satisfaction  with  Proposal  Feedback  Time 


Q-3  What  time  standard  should  NASA  use  for  providing  proposal  feedback? 


Desired  time  standard  for  proposal  feedback  (months) 


Figure  1-3. 

Desired  Feedback  Time 
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Q-4  What  has  been  your  actual  time,  on  the  average,  for  receiving  proposal  feed¬ 
back? 


Actual  proposal  feedback  time  (months) 


Figure  1-4. 

Actual  Feedback  Time 


Q-5  How  satisfied  are  you  with  length  of  the  grant  period  for  executing  re¬ 
search? 


Figure  1-5. 

Satisfaction  with  Length  of  Grant  Period 
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Q-6  What  period  of  time  would  you  like  to  see  for  executing  research? 


Figure  1-6. 

Desired  Grant  Period 


Guidelines 


Q-7  How  satisfied  are  you  with  the  overall  clarity,  simplicity,  and  completeness 
of  the  NASA  Research  Announcement  (NRA)? 


Level  of  satisfaction 


LEGEND: 

1  =  Not  Satisfied 

2  =  Slightly  Satisfied 

3  =  Fairly  Satisfied 

4  =  Very  Satisfied 

5  =  Extremely  Satisfied 


Figure  1-7. 

Satisfaction  with  NRAs 
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Q-8  How  satisfied  are  you  with  key  instructions  covered  in  the  NRA? 


Figure  1-8. 

Satisfaction  with  Instructions  —  Program  Scope 


Level  of  satisfaction 


LEGEND: 

1  =  Not  Satisfied 

2  =  Slightly  Satisfied 

3  =  Fairly  Satisfied 

4  =  Very  Satisfied 

5  =  Extremely  Satisfied 


Figure  1-9. 

Satisfaction  with  Instructions  —  Proposal  Contents 


1-7 


LEGEND: 

1  =  Not  Satisfied 

2  =  Slightly  Satisfied 

3  =  Fairly  Satisfied 

4  =  Very  Satisfied 

5  =  Extremely  Satisfied 


Level  of  satisfaction 


Figure  1-10. 

Satisfaction  with  Instructions  —  Preparation  and  Submission 


LEGEND: 

1  =  Not  Satisfied 

2  =  Slightly  Satisfied 

3  =  Fairly  Satisfied 

4  =  Very  Satisfied 

5  =  Extremely  Satisfied 

1  2  3  4  5 


Level  of  satisfaction 


Figure  1-11. 

Satisfaction  with  Instructions  —  Evaluation  Factors 
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Percent 

50] 


Figure  1-12. 

Satisfaction  with  Instructions  —  Costing 


General 


Q-9  How  adequate  is  NASA  feedback  on  your  proposals? 


Figure  1-13. 

Adequacy  of  Feedback 
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Q-10  How  satisfied  are  you  that  NASA  fairly  evaluates  your  research  proposals? 


Level  of  satisfaction 


LEGEND: 

1  =  Not  Satisfied 

2  =  Slightly  Satisfied 

3  =  Fairly  Satisfied 

4  =  Very  Satisfied 

5  =  Extremely  Satisfied 


Figure  1-14. 

Satisfaction  with  Evaluation  Fairness 


Q-ll  Adequate  feedback,  fairness,  processing  time,  and  processing  guidance  are 
essential  elements  of  the  NASA  science  grant  program.  How  important  to 
•  you  is  NASA's  feedback,  fairness,  processing  time,  or  process  guidance? 


Figure  1-15. 

Importance  of  Proposal  Feedback 
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Percent 
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78.25% 


2  3  4  5 


LEGEND: 

1  =  Not  Important 

2  =  Slightly  Important 

3  =  Fairly  Important 

4  =  Very  Important 

5  =  Extremely  Important 


Level  of  importance 


Figure  1-16. 

Importance  of  Evaluation  Fairness 


Figure  1-17. 

Importance  of  Processing  Time 
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Figure  1-18. 

Importance  of  Process  Guidance 


Q-12  How  satisfied  are  you  with  the  overall  NASA  science  grant  process? 


Figure  1-19. 

Satisfaction  with  Overall  Grant  Process 


1-12 


Astropartiue  physics 

1 

2.20 

Atmospheres 

1 

2.20 

Atmospheric  science 

1 

2.20 

Biology 

2 

4.30 

Computer  science 

1 

2.20 

Cosmology 

4 

8.80 

Exobiology 

1 

2.20 

Fluid  mechanics 

1 

2.20 

Geochemistry 

1 

2.20 

Geodynamics 

1 

2.20 

Geology 

1 

2.20 

Geophysics 

1 

2.20 

Geoscience 

1 

2.20 

Gravitation 

1 

2.20 

Human  factors 

1 

2.20 
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Table  1-1. 

Other  Disciplines  (Continued) 


Category 

Frequency 

Percentage 

Impact 

1 

2.20 

Laboratory  astrophics 

1 

2.20 

Laboratory 

1 

2.20 

Lasers,  atmosphere 

1 

2.20 

Life  sciences 

1 

2.20 

Mathematics 

1 

2.20 

Meteorite  impact 

1 

2.20 

Meteorites 

1 

2.20 

Meteoritics 

i 

2.20 

Physical  chemistry 

1 

2.20 

Planetary 

1 

2.20 

Planetary  atmosphere 

3 

6.50 

Planetary  geology 

1 

2.20 

Planetary  material 

3 

6.50 

Planetary  science 

1 

2.20 

Quantum  devices 

1 

2.20 

Remote  scusing 

1 

2.20 

Solar  physics 

2 

4.30 

Solar  terrestrial 

1 

2.20 

Space  chemistry 

1 

2.20 

Terrestrial  plan 

2 

4.30 

Total 

46 

100.00 
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Table  1-2. 

Subdisciplines 


Category 

Frequency 

Percentage 

Active  galaxies 

1 

0.35 

Aeronomy 

2 

0.70 

X-Ray 

1 

0.35 

Applied  mathematics 

1 

0.35 

Artificial  intelligence 

1 

0.35 

Asteroids 

1 

0.35 

Astronomy 

2 

0.70 

Astrophysics 

1 

0.35 

Atmospheres 

5 

1.80 

Atomic  and  molecular 

5 

1.80 

Bioengineering 

1 

0.35 

Suborbita 

1 

0.35 

Chemistry 

2 

0.70 

Chromospheric 

1 

0.35 

Close  binary 

1 

0.35 

Cometary  research 

2 

0.70 

Compact  objects 

1 

0.35 

Computational 

1 

0.35 

Cosmic  heliosphere 

1 

0.35 

Cosmic  rays 

1 

0.35 

Cosmochemistry 

5 

1.80 

Data  analysis 

2 

0.70 

Detectors 

1 

0.35 

Dynamics  and  galaxy 

3 

1.10 

Dynamos 

1 

0.35 

Early  evolution 

1 

0.35 

Electrical  engine 

1 

0.35 

Engineering  science 

1 

0.35 

Euv  astronomy 

1 

0.35 

Exobiology 

4 

1.40 

Experimental  geology 

1 

0.35 

Extragalactic 

3 

1.10 

Extrasolar 

1 

0.35 

Extraterrestrial 

1 

0.35 

Formation  of  proteins 

1 

0.40 

Galactic  structures 

2 

0.70 
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Table  1-2. 

Subdisciplines  (Continued) 


Category 

Frequency 

Percentage 

Gamma  ray  bursts 

1 

0.35 

General  relativity 

1 

0.35 

Geochemistry 

2 

0.70 

Geology 

5 

1.80 

Geophysics 

2 

0.70 

Geophysics  (solid) 

1 

0.35 

Geophysics  (plane) 

1 

0.35 

Heliospheric 

2 

0.70 

High  energy 

6 

2.20 

Hot  stars 

1 

0.35 

Impact  geophysics 

1 

0.35 

Infrared/Star  formation 

1 

0.35 

Instrumentation 

2 

0.70 

Iterstellar 

8 

2.90 

Ion-molecule 

8 

2.85 

Ionosphere 

4 

1.40 

Spectroscope 

1 

0.35 

Laboratory 

5 

1.80 

Lidar 

1 

0.35 

Lunar  dynamics 

1 

0.35 

Magnetic  fields 

2 

0.70 

Magnetosphere 

24 

8.55 

Massive  star  formations 

1 

0.35 

Materials 

2 

0.70 

Mesosphere 

2 

0.70 

Meteritics 

1 

0.35 

Molecular  evolution 

1 

0.35 

Molecular  spectrum 

2 

0.70 

Nuclear  planetary 

1 

0.35 

Observational 

2 

1.70 

Old  stellar  system 

1 

0.35 

Origins 

2 

0.70 

Particle  cosmology 

1 

0.35 

Particle  precipitation 

1 

0.35 

Petrology 

1 

0.35 

Physics 

1 

0.35 

Planetary  astronomy 

7 

2.50 

Planetary  atmosphere 

6 

2.10 
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Table  1-2. 

Subdisciplines  (Continued) 


Category 

Frequency 

Percentage 

Planetary  geology 

12 

4.30 

Planetary  material 

12 

4.30 

Planetary  physics 

2 

0.70 

Planetary  science 

1 

0.35 

Planetary  surface 

3 

1.10 

Plasma  astrophysics 

1 

0.35 

Quasars 

1 

0.35 

Radio  and  X-ray 

2 

0.70 

Relativity 

1 

0.35 

Remote  sensing 

1 

0.35 

Software  development 

1 

0.35 

Solar  physics 

18 

6.40 

Solar  wind 

2 

0.70 

Space  physics 

3 

1.10 

Spectroscopy 

2 

0.70 

Star  dust 

1 

0.35 

Star  formation 

1 

0.35 

Stars  and  galactics 

2 

0.70 

Stellar  research 

7 

2.50 

Stratospheric 

1 

0.35 

Submillimeter 

1 

0.35 

Sububatal 

1 

0.35 

Supernova 

2 

0.70 

The  verticale 

1 

0.35 

Theoretical  astronomy 

5 

1.80 

Theory 

6 

2.10 

Theory  —  high  energy 

2 

0.70 

Theory  and  space 

2 

0.70 

Therm  osphere/ion 

1 

0.35 

Unspecified 

1 

0.35 

Upper  atmosphere 

1 

0.35 

UV  and  X-ray 

1 

0.35 

UV  astronomy 

1 

0.35 

UV  astrophysics 

1 

0.35 

UV  from  space 

1 

0.35 

Uvoir 

1 

0.35 

Vo  Jean  ism 

1 

0.35 

Wave  phenomena 

1 

0.35 
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Table  1-2. 

Subdisciplines  (Continued) 


Category 

Frequency 

Percentage 

X-ray  astronomy 

1 

0.35 

X-ray  pulsars 

1 

0.35 

Total 

279 

100.00 

Q-14  How  many  years  have  you  submitted  NASA  grant  proposals? 


Proposal  experience  (years) 

Figure  1-21 . 

Years  of  Proposal  Submissions 
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Q-15  What  percentage  of  your  proposal  submissions  have  been  selected? 


<20  21-40  41-60  61-80  >80 

Proposal  success  rate  (percent) 


Figure  1-22. 

Percentage  of  Proposals  Selected 
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Survey  Comments: 
Proposal  Writers 


Survey  Comments:  Proposal  Writers 


In  this  appendix,  we  quote  the  comments  made  by  the  proposal  writers  on 
the  science  grant  survey.  (The  comments  have  been  edited  slightly  for  clarity, 
brevity,  and  grammar.)  We  have  organized  the  comments  according  to  the  sub¬ 
sections  in  the  survey: 


Subsection  Page 

Time  . J-3 

Guidelines  . J— 1 1 

General  . J-20 


Time 


♦  Speed  up  evaluation. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  have  always  felt  left  in  the  dark  about  whether  or  not  my  grant  will  be 
funded  because  I  have  never  received  any  information  in  six  months.  It 
takes  several  inquiries  on  my  part  to  get  an  answer  and  further  pestering  by 
our  business  office  to  actually  secure  the  funds.  This  is  annoying. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Reviews  occur  in  November,  yet  no  results  are  known  until  late  January  or 
early  February.  This  is  presumably  due  to  budget  decisions.  Otherwise  one 
would  assume  that  the  decisions  could  be  made  immediately  after  the  panel 
reviews.  If  there  is  always  going  to  be  this  wait,  due  to  budget  decisions, 
why  not  move  the  proposal  writing  process  to  a  later  time  so  that  the  two 
would  be  better  synchronized? 

Once  an  award  is  made,  the  delay  in  actually  getting  the  money  to  an  insti¬ 
tution  has  crept  up  to  an  additional  2-3  months.  If  you  take  3  months  for 
proposal  writing,  6  months  for  decisions  and  2-3  months  to  get  the  money, 
this  process  lasts  an  entire  year.  This  is  too  long! 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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[Make]  prompt  decisions  (six  months  or  less)  regarding  funding. 

The  following  comments  are  designed  to  reduce  the  amount  of  time  spent 
preparing  proposals: 

►  Establish  principal  investigators  (Pis)  (and  perhaps  others,  although 
they  might  not  wish  to  risk  it)  [who  have]  the  option  of  doing 
performance-based  proposals,  much  as  is  done  by  NIH. 

►  Longer  grant  periods  than  the  current  three  years  should  be  possible. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

Less  delay  in  [receiving]  funding,  i.e.,  once  a  commitment  is  made,  the  start 
should  be  as  fast  as  possible. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦. 

Right  now  perhaps  the  biggest  problem  is  the  delay  between  positive  notifi¬ 
cation  of  a  grant  award  and  the  time  the  money  reaches  the  researcher's  in¬ 
stitution.  The  current  situation  is  unacceptable! 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

The  [time]  standards  themselves  are  reasonable,  but  the  actual  [times]  are 
much  longer  for  NASA  HQ. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

In  view  of  the  effort  required,  a  grant  period  of  a  minimum  of  2  years  would 
permit  more  science  and  less  paperwork  to  be  completed. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

The  key  areas  are  a  more  timely  announcement  schedule  [and]  a  more 
timely  schedule  for  evaluation  and  feedback.  If  deadlines  slip  on  feedback, 
the  grantee  should  not  be  penalized  by  having  to  do  a  three-year  project  in 
less  [time]. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

My  only  complaint  is  the  relative  delay  in  actually  starting  the  grant.  In  my 
case,  I  had  to  contact  the  headquarters  6  weeks  past  the  starting  date  to  get 
things  going. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

Feedback  and  communication  need  much  improvement!  Proposers  need  to 
know  (1)  when  their  proposal  has  been  received,  (2)  when  the  reviews  will 


be  completed,  and  (3)  when  they  will  be  notified  one  way  or  the  other  if  they 
will  be  funded.  Program  scientists  need  to  be  more  easily  accessible  to  an¬ 
swer  questions.  But  again,  the  most  important  improvement  would  be  bet¬ 
ter,  faster,  cheaper,  feedback  and  communication. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Decisions  on  proposal  selection  should  be  communicated  to  proposer  by  the 
start  of  the  fiscal  year!!  4  to  5  months  should  be  sufficient  for  the  reviewing 
process! 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Fund  proposals  for  3  years. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Allow  for  longer  term  automatic  data  processing  (ADP)  grants.  Right  now 
there  are  the  extremes  of  either  1  year  ADP  or  5  year  LTSA  and  nothing  in 
between.  3  year  grants  would  be  very  useful,  particularly  for  hiring  post¬ 
doctorate  researchers. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  [Provide]  longer  grant  periods.  Shorten  times  for  proposal  feedback. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  submitted  a  proposal  in  December  1992.  In  December  1993, 1  got  the  "con¬ 
gratulations"  (acceptance)  letter.  During  the  spring  of  1994  I  was  harassed 
twice  to  provide  information  which  NASA  already  had.  In  January  1995 
(!!!!)  the  funds  could  be  spent.  Total  grant  money  [was]  about  $7,500.00 

Conclusion:  create  a  category  of  "trial"  grants,  funding  less  than  $10,000, 
one  year  at  the  time,  with  very  fast  turnaround  time  (from  submission  to  acti¬ 
vation  —  in  six  months)  An  investigator  can  have  only  one  such  grant. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  main  problem  I  have  had  with  the  process  is  after  approval  —  it  usually 
takes  6  months  or  more  to  get  the  funding  going,  and  in  one  recent  [case]  it 
was  over  a  year  from  [time  of]  approval  to  the  start  of  funding! 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  [announcement]  should  be  released  earlier  with  2  months  allowed  for 
proposal  preparation.  This  would  enable  NASA  to  announce  winners  by 
the  end  of  November  and  complete  the  grants  early  in  the  [calendar  year]. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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♦  When  proposals  seem  well  thought  out  to  sequence  of  study,  fund  for  the 
full  period  (2-5  years)  without  requiring  annual  [reviews]. 

Don't  put  out  announcements  for  things  that  are  not  almost  certain  to  be 
funded!!! 

Do  reviews  and  make  decisions  in  <  2  months.  Have  review  panels  in  place 
by  the  time  proposals  are  due. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  My  chief  concern  is  the  time  allotted  for  preparing  a  proposal.  At  universi¬ 
ties  our  time  commitments  vary  widely  during  the  year  depending  on  when 
we  teach.  If  the  3-month  proposal  preparation  period  overlaps  a  teaching 
commitment,  we  are  at  a  disadvantage.  Just  increasing  the  3-month  period 
to  4  months  would  be  a  great  improvement. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Although  proposals  are  nominally  funded  for  three  years,  in  fact  the  amount 
of  time  to  actually  perform  research  based  on  those  funds  is  somewhat 
shorter  due  to  inherent  time  delays  in  the  system.  Frequently  one  has  to 
start  writing  a  renewal  proposal  well  before  the  current  proposal  has  ex¬ 
pired.  This  is  particularly  taxing  when  trying  to  initiate  a  new  line  of  re¬ 
search.  One  possible  alternative  would  be  to  fund  proposals  for  three  years 
with  two  one-year  renewals  that  are  granted  on  the  basis  of  successful  pro¬ 
gress  as  evidenced  by  short  progress  reports.  The  project  could  be  termi¬ 
nated  at  the  end  of  the  third  or  fourth  year,  if  progress  is  not  satisfactory, 
but  a  full  renewal  proposal  would  not  be  required  until  the  end  of  the  fifth 
year. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  A  recurrent  problem  has  been  the  length  of  time  between  notification  of  re¬ 
view  results  and  actual  funding  (ranging  from  4  to  6  months).  If  funding 
must  be  delayed,  then  the  start-up  date  should  be  corrected  to  match  reality! 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Appear  to  get  comments  back  on  proposals  in  reasonable  time  but  some¬ 
times  [it  takes]  more  than  a  year  from  proposal  submission  to  when  money 
comes.  This  varies  by  [a  factor  of  2  to  3]  depending  on  the  year. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  A  major  problem  is  the  time  required  to  actually  obtain  funds.  This  is  the 
largest  "bottleneck"  in  the  system.  It  took  1.5  years  for  me  to  obtain  funds 
for  my  most  recent  award:  6  months  for  the  review  and  1  year  for  the  con¬ 
tracts  office  to  actually  award  the  funds. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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♦  Make  decisions  [as  soon  as  possible]  after  review  panels  meet,  and  promptly 
inform  Pis. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  present  low  probability  of  success  (approximately  20  percent)  for  pro¬ 
posals  requires  Pis  to  submit  new  proposals  essentially  every  year  in  order 
to  maintain  funding  continuity.  If  one  were  to  wait  until  the  year  a  funded 
proposal  is  to  expire,  one  would  have  a  very  high  probability  of  being  un¬ 
funded  for  at  least  [a]  year.  As  a  result.  Pis  write  many  more  proposals  than 
are  necessary,  thus  consuming  much  time  and  effort  for  both  the  PI  and  the 
reviewers.  I  recommend  that  proposals  be  funded  for  longer  periods  and  be 
more  easily  renewed,  if  good  progress  is  being  made.  In  return,  the  PI 
should  be  restricted  from  applying  for  more  funds  during  the  period  for 
which  the  PI  is  funded. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  time  between  proposal  submission  and  funding  should  be  rigorously 
restricted  to  under  7  months. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  [Provide]  more  rapid  feedback  on  proposals. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  overriding  priority  should  be  to  speed  up  the  process  of  actually  send¬ 
ing  money  after  the  award  letter.  In  some  cases  that  has  taken  as  long  as 
9  months,  which  means  15  - 16  months  after  the  submission  of  a  proposal. 
It  is  unfair  to  universities  to  expect  them  to  "backstop"  all  of  a  Pi's  research 
expenses  for  that  long  on  the  basis  of  a  nonbinding  number. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Full  proposals  consume  too  much  time  in  a  professional  career  —  especially 
when  the  outcome  is  unknown.  Perhaps  it  would  be  better  to  have  brief 
(4  page)  pre-proposals  submitted  that  the  Program  Manager  can  filter  and 
say  "go  ahead,  provide  a  full  proposal,"  or  "this  is  not  appropriate  —  don't 
waste  any  more  time  on  it."  Combine  this  with  a  rapid  review  process 
(1  month),  and  it  would  enormously  streamline  the  system  for  applicants,  - 
managers,  and  reviewers  alike. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Government  laboratory  budgets  are  centered  around  the  fiscal  year.  Even 
university  budgets  for  laboratories  may  run  on  a  semester  schedule.  Our  lab 
(government)  is  making  financial  commitments  beginning  in  October  and 
operations  would  be  a  lot  smoother  if  results  from  early  summer  proposal 
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submission  were  known.  If  it  really  takes  6  months  from  submission  to  noti¬ 
fication  then  why  not  move  submission  dates  back  into  March  or  April? 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  In  my  experience  it  takes  too  long  after  a  grant  has  been  approved  by  the  re¬ 
view  panel  for  the  funds  to  be  actually  received. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  '  Time  line  is  too  long  —  one  proposal  took  14  months  before  I  [received]  the 

first  feedback.  This  was  in  spite  of  numerous  attempts  to  contact  the  direc¬ 
tor  for  information.  A  second  proposal  was  evaluated  in  4  months,  but  it  is 
now  [an  additional]  5  months  and  I  have  not  heard  anything  concerning  the 
actual  funding  level. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Fund  for  at  least  2  years  at  a  time  rather  than  every  year. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  funding  process  beyond  HQ  can  only  be  described  as  ludicrous  and  ab¬ 
surd.  [Six  to  nine]  months  to  [receive]  the  funding  is  inefficient. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Some  proposals  have  taken  more  than  a  year  (years!)  to  get  a  response. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  time  it  takes  between  announcement  of  an  award  and  the  award  is  ex¬ 
cessive,  even  with  continuation  awards.  Most  of  the  delay  seems  to  be  in  the 
grants  office.  Often  the  time  between  the  NASA  research  announcement 
(NRA)  and  the  deadline  is  too  short. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  grant  period  should  be  for  three  years.  While  three  years  is  allowed  as  a 
grant  period,  I  hesitate  to  write  grants  for  that  period  because  the  descrip¬ 
tion  of  research  covering  the  period  must  necessarily  be  vague,  and  vague 
descriptions  do  not  get  good  reviews.  It  would  be  nice  if  this  perception 
could  somehow  be  removed  from  the  proposal  process. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 


♦  [The]  proposal  process  needs  to  be  run  in  a  more  timely  fashion,  so  that 
awarded  funds  are  available  at  [the]  beginning  of  [the]  financial  year  and 
everyone  can  work  to  a  common  timetable. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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♦  Funding  periods  need  to  be  long  enough  so  that  there  is  time  to  do  research, 
and  not  just  proposal  writing. 

There  is  a  serious  problem  also  in  that  NASA  has  created  a  highly  skilled 
group  of  professional  scientists  in  soft-money  positions  who  do  much  of 
NASA's  science,  yet  are  highly  vulnerable  to  fluctuations  in  the  grant  proc¬ 
ess.  Rapid  review  of  proposals  —  and  rapid  funding  of  approved 
proposals  —  is  vital  to  the  survival  of  this  group  of  scientists. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  NRA  process  for  major  programs  usually  provides  an  excellent  oppor¬ 
tunity,  but  with  too  little  time  allowed  for  preparation  and  subsequent  delay 
in  carrying  out  the  successful  proposals  (if  they  ever  do  get  carried  out). 

The  least  successful  aspect  of  the  grants  process  is  the  mechanism  for  pro¬ 
viding  funds  (the  Grants  office).  This  is  incredibly  slow  and  unresponsive. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  worst  delays  are  at  NASA  HQ,  where  decisions  take  much  too  long  for 
many  programs.  Getting  funds  to  Pis  is  getting  better,  although  delays,  [re¬ 
sulting  from]  revised  budgets,  have  meant  that  funds  arrive  almost  a  year 
after  a  proposal  was  submitted. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  A  major  problem  is  the  long  and  unpredictable  delay  between  notification  of 
a  successful  proposal  and  receipt  of  the  money.  This  can  make  planning  and 
executing  a  research  program  very  difficult. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  My  biggest  concern  is  the-lack  of  year-round  proposal  and  funding  opportu¬ 
nities.  There  should  be  at  least  some  program  under  which  investigators 
can  apply  for  modest  funding  at  any  time  and  have,  if  successful,  some 
funds  within  4  to  6  months  of  submission. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  single  [grant]  processing  problem  is  the  enormous  length  of  time  be¬ 
tween  submitting  a  proposal  and  finally  getting  funding.  I  get  a  fairly  rapid 
response,  [less  than]  6  months,  but  then  the  waiting  begins  —  typically 
12  months  or  more  between  a  positive  response  and  the  arrival  of  funding. 
Meanwhile,  I  have  to  scramble  to  cover  salaries,  etc.  This  is  an  enormous 
source  of  frustration  with  the  NASA  system,  against  which  all  others  pale  in 
comparison. 
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♦  Probably  the  two  most  important  items  to  improve  on  (in  my  mind)  are: 


►  Quicker  response  to  evaluation  results 

►  Longer  grant  periods  viewed  more  acceptably 

Both  would  allow  better  planning  on  the  part  of  investigators.  As  things 
stand  now,  many  of  us  have  no  clue  what  we  will  be  able  to  do  in  the  next 
6-12  months.  This  is  especially  true  if  funding  is  sought  to  support  the  re¬ 
search,  graduate  students,  etc. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Allow  flexibility  (e.g.,  5-year  grants;  3-year  grants;  1-year  grants; 
small/ short  grants). 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  New  starts  are  very  difficult.  By  the  time  I  got  the  first  money  (approxi¬ 
mately  1  year  later)  it  was  time  to  submit  a  new  proposal. 

I  had  one  proposal  that  stayed  in  limbo  for  2  years  because  of  program  cuts 
and  delays.  Try  not  to  let  this  happen. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  review  process  takes  too  long.  I  have  waited  more  than  a  year  for  a  re¬ 
sponse  to  a  proposal,  during  which  time  the  Program  Manager  did  not  re¬ 
turn  my  phone  calls  or  E-mail  messages. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  have  no  complaints  about  the  proposal  process,  but  I  do  have  about  the 
grant  process,  i.e.,  after  the  proposal  is  awarded.  In  my  case,  grant  monies 
invariably  arrive  after  the  date  I  have  been  told  to  expect  them.  In  the  last 
go-around  the  money  I  was  to  have  received  on  April  1, 1994,  did  not  arrive 
until  January  15,  1995!  And  the  money  that  was  to  have  arrived  on 
October  1,  1994,  has  not  arrived  yet.  This  is  grossly  unfair  since  it  is  very 
difficult  to  plan  this  way.  The  university  has  been  generous  in  loaning  me 
the  promised  funds  and  NASA  actually  delivers,  but  this  is  of  course  not 
cost  free  to  them  either.  Why  does  the  grant  administration  have  to  be  so 
"messed  up"? 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  feedback  on  results  is  OK,  but  the  response  time  [for  receiving  funds]  is 
often  6-9  months! 
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♦  The  major  problems  occur  after  the  grant  award.  Delays  in  funding  cause 
major  cash  flow  dislocations.  Sometimes  funding  is  delayed  for  6-12  months 
in  the  middle  of  a  multi-year  grant.  Beside  making  appointments  uncertain, 
this  delays  funding  into  the  one-time  no  cost  extension  period.  After  the 
mid-grant  delays  this  forces  rapid  and  possibly  inefficient  disbursement  of 
funds  to  meet  file  hard  deadline.  If  funding  is  delayed  by  NASA,  the  grant 
anniversary  dates  should  be  adjusted  to  leave  some  flexibility  in  disburse¬ 
ment. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Of  course  it  would  be  great  if  NASA  could  process  and  award  grants  in  less 
than  6  months.  However,  given  current  funding  and  personnel  limitations 
at  NASA,  they  are  lucky  to  meet  the  6  month  guideline. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  major  concern  that  I  have  with  NASA's  science  grant  process  is  that  too 
often  the  evaluations  indicate  that  the  peer  review  panel  did  not  understand 
the  proposal  very  well.  This  is  acceptable  when  the  proposal  is  poorly  writ¬ 
ten,  but  too  often  well-written  proposals  are  not  properly  evaluated  because 
the  panel  is  given  too  many  proposals  to  review.  I  know  of  panels  consist¬ 
ing  of  8  to  10  people  which  are  given  100  or  more  proposals  to  evaluate.  In 
this  case,  each  person  on  the  panel  may  be  primary  or  secondary  reviewer 
for  about  20  proposals,  which  requires  more  time  than  most  of  us  have. 
Many  panel  members  then  do  not  have  the  time  to  read  carefully  the  re¬ 
maining  80  or  so  proposals  that  they  are  asked  to  vote  on.  As  a  result,  panel 
members  often  do  not  understand  the  proposals  in  sufficient  detail  and  the 
ranking  and  evaluation  suffers  accordingly. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  We  all  spend  an  awful  lot  of  time  writing  and  reviewing  so  many  proposals. 
Perhaps  a  move  towards  fewer,  larger  grants  is  in  order.  The  reviewing/ 
processing  overhead  perhaps  outweighs  the  flexibility  we  have  for  funding 
one  person  for  a  month  or  two  per  year. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 


Guidelines 

♦  Sometimes  I  find  the  instructions  confusing  because  to  understand  all  the  re¬ 
quirements  one  needs  to  flip  back  and  forth  between  the  general  instructions 
in  the  Appendices  and  specific  instructions  in  the  body  of  the  announce¬ 
ments. 
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♦  NASA  grants  have  requirements  which  are  not  known  to  Discipline  Scien¬ 
tists  (DS)  in  Washington.  I  do  what  the  DS  says  and  it  is  often  contradictory 
to  the  official  document.  The  most  flagrant  sample  is  that  NASA  grants  re¬ 
quire  reports  whereas  the  DS  considers  the  progress  report  section  of  the  next 
proposal  as  the  Report. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  For  accepted  proposals,  funding  decisions  based  on  project  needs  rather 
than  mindless  formulas  based  on  the  previous  year's  budget  (and  possible 
COLAs,  when  the  PI  is  fortunate) 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  All  reviews  should  be  returned  to  proposers  rather  than  summaries  pre¬ 
pared  by  HQ  employees. 

Proposers  should  have  an  opportunity  to  respond  to  review  comments  and 
this  information  should  be  available  to  panel  review  teams. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Rely  on  written  reviews  more. 

Emphasize  results  rather  than  NASA  programmatic  issues. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  All  the  programs  should  let  people  know  through  the  NRA  what  supple¬ 
mental  material  is  available  to  help  the  proposal  writers  evaluate  whether 
their  research  is  relevant  to  the  program.  I  have  received  reports  of  working 
groups,  study  groups,  etc.,  as  supplemental  material. 

-  ♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Greater  clarity  as  to  whether  specific  topics  are,  or  are  not,  appropriate  to 
operate  NRAs. 

[Provide]  more  realistic  evaluation  sheets:  if  a  proposal  is  rejected  yet  the 
number  of  "pros"  exceeds  the  number  of  "cons"  and  the  "cons"  seem  arbi¬ 
trary  or  irrelevant,  it  is  hard  to  understand  the  inherent  message. 

The  measure  of  the  adequacy  of  the  review  process  is  best  manifest  in  the 
skill  exercised  in  designing  an  evaluation  sheet  especially  when  a  proposal 
has  been  rejected.  Often  evaluation  sheets  are  terse  or  cryptic.  How  can  one 
learn  if  not  through  these  words? 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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♦  Perhaps  an  early  feedback  on  the  proposal  would  allow  any  gross  misun¬ 
derstandings  to  be  corrected  before  final  selection. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Adopt  and  publicize  a  reasonable  set  of  guidelines  for  conflict  of  interest 
and  follow  them. 

The  last  NRA  to  which  I  responded  had  conflicting  requirements  in  different 
portions  of  the  document.  Please  simplify  it,  and  get  it  straight. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  [Establish]  clear  evaluation  criteria. 

Open  access  to  results  so  proposers  would  know  what  proposals  are  se¬ 
lected. 

♦  ♦♦♦♦♦♦♦♦♦.♦♦♦♦ 

♦  Oftentimes  not  even  all  of  the  straight  "excellent"  proposals  are  funded. 
Conveys  a  sense  of  capriciousness  to  the  entire  process.  This  puts  pressure 
on  the  Pis  to  stagger  their  proposals,  submitting  a  small  proposal  every  year 
rather  than  a  larger  one  every  two-three  years,  in  the  hopes  of  getting  at 
least  partial  support.  This  puts  further  pressure  on  the  programs  and  aggra¬ 
vates  the  problem. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  NRAs  can  be  very  turgid  —  a  summary  sheet  of  requirements  (in  the 
form  of  a  check-off  list  perhaps)  would  speed  the  preparation  as  would  an 
index  and  clear  presentation  of  the  sections.  At  the  moment  it  seems  that 
NRAs  are  in  transition  from  "all  paper"  to  electronic,  and  putting  all  the  pa¬ 
perwork  together  takes  a  great  deal  of  time  and  [effort], 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  NASA  should  publish  a  list  of  successful  proposals  for  every  competition. 

I  don't  like  the  nebulous  "programmatic  factors"  NASA  uses  to  make  selec¬ 
tions.  In  too  many  cases,  it  seems  the  review  panel's  recommendations  are 
ignored  —  weaker  proposals  are  funded  while  excellent  proposals  are  not. 
I'd  like  to  see  science  be  the  determining  factor. 

I  dislike  rules  imposed  after  the  fact.  [In]  some  years,  decisions  were  made 
during  the  review  process  not  to  fund  anyone  who  already  had  support. 
What  a  waste  of  time  and  effort  for  those  who  [prepared]  proposals  and 
were  summarily  dismissed! 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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♦  Program  scope  described  in  NRAs  is  often  not  consistent  with  what  propos¬ 
als  are  actually  invited  or  funded. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Send  out  self-contained  NRAs  that  contain  all  the  necessary  forms;  (now  we 
have  to  [go  through]  many  different  files  and  piece  together  research  infor¬ 
mation). 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  scope  of  research  described  in  the  NRA  is  not  always  accurate.  Talking 
to  NASA  sometimes  clarifies  things  quite  a  lot. 

Unfortunately,  there  is  not  always  a  scientist  to  speak  to,  and  the  true  scope 
is  difficult  to  see. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Colleagues,  within  and  outside  NASA,  often  complain  that  the  program 
scope  as  stated  in  the  NRAs  changes  after  the  proposals  are  submitted,  mak¬ 
ing  some  of  the  submitted  proposals  irrelevant  to  the  program,  a  very  waste¬ 
ful  situation.  Program  directors  should  be  given  the  authority  to  decide 
such  cases,  consistent  with  the  fact  that  they  are  scientists  who  are  compe¬ 
tent  in  the  program  discipline  and  aware  of  the  NASA  needs. 

Writing  a  cost  management  plan  is  difficult  for  most  research  efforts,  in 
which  it  is  difficult  to  gauge  direction  more  than  a  few  weeks  at  a  time.  This 
may  be  best  reserved  for  large  project  team  proposals. 

The  three  month  period  given  to  respond  to  an  NRA  rules  out  the  participa¬ 
tion  [of]  scientists  not  currently  in  the  information  loop.  Such  scientists  and 
NASA  may  be  missing  great  opportunities. 

NASA  (the  government)  must  abandon  the  requirement  that  funds  be  used 
within  one  year  for  each  funded  year.  Costs  can  often  be  reduced  if  ex¬ 
penses  can  be  postponed.  Moreover,  a  grantee  should  not  be  penalized  if 
he/she  finds  a  way  of  doing  the  same  project  with  less  funds  than  originally 
proposed.  The  grantee  should  be  allowed  to  set  aside  remaining  funds  to  be 
used  the  following  year. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  NRA  is  poorly  organized  —  one  does  not  know  where  to  look  for  cer¬ 
tain  pieces  of  information.  In  places  it  repeats  itself  partially.  The  reader 
then  doesn't  know  if  the  statement  he  just  read  is  the  most  accurate  version 
of  the  statement  in  the  NRA. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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Remove  graduate  student  participation  as  a  selection  criteria. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

[The]  call  for  proposals  and  [the]  instructions  are  not  bad  —  they  are  nearly 
in  English!  Could  be  clearer. 

The  program  scope  is  often  ill  defined,  especially  for  NRAs  that  are  not  part 
of  a  series,  such  as  the  ADP  series.  In  particular,  the  line  between  paper 
studies  and  hardware  development  is  often  left  unclear  in  the  NRA.  Re¬ 
peated  calls  to  HQ  result  in  wildly  different  interpretations.  Since  hardware 
development  often  drives  mission  concept  and  design,  it  should  be  given  a 
more  prominent  place  within  NASA's  grant  program,  and  given  a  clearer 
demarcation  within  individual  NRAs. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

The  PI  should  be  given  the  opportunity  to  respond  to  comments  before  a 
final  decision  is  made,  especially  for  borderline  cases. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

I  would  prefer  that  the  proposals  for  a  given  NRA  be  divided  among  a 
larger  number  of  peer  review  panels,  so  that  each  panel  is  given  a  smaller 
number  of  proposals  to  review.  These  smaller  panels  will  have  less  exper¬ 
tise,  but  with  fewer  proposals  to  review,  the  smaller  panels  can  be  more  fo¬ 
cused  on  the  science  topic  and  do  a  better  job.  Merging  the  relative  rankings 
of  a  larger  number  of  panels  for  the  allocation  of  scarce  funding  is  probably 
best  done  by  just  selecting  the  highest  ranked  proposals  in  each  panel.  The 
usual  technique  of  asking  the  panel  chairs  to  compare  the  rankings  of  each 
panel  is  often  flawed,  because  comparisons  are  often  done  in  haste  without 
the  panel  chairs  having  the  opportunity  (or  in  some  cases  the  expertise)  to 
read  and  understand  the  proposals  reviewed  by  the  other  panels. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

More  uniformity  should  be  enforced  on  current  and  pending  support  state¬ 
ments  OR  they  should  be  eliminated. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

The  proposer  should  have  an  opportunity  for  rebuttal  before  a  final  decision 
is  reached. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

The  format  of  the  proposals  themselves  could  be  streamlined.  Eliminate  the 
redundant  and  pedantic  "Budget  Summary"  pages.  Obtain  a  single  annual 
copy  of  the  drug-free  workplace,  etc.,  forms  from  each  university  instead  of 
the  wasteful  duplication  of  putting  them  in  every  proposal.  The  forms  are 


signed  by  a  representative  of  the  whole  university,  not  the  PI:  either  the  uni¬ 
versity  is  in  compliance  or  it  isn't;  there's  no  need  to  repeat  that  statement  in 
every  proposal. 

Improvements  can  be  made  in  the  area  of  grant  administration.  "Nitpicky" 
challenges  to  budget  items  from  the  grants  office  should  stop.  It  is  ridicu¬ 
lous  to  require  travel  destinations,  airfare  and  hotel  costs,  and  which  journal 
page  charges  refer  to,  when  these  items  are  a  tiny  fraction  of  a  budget,  and 
the  budget  had  to  be  prepared  more  than  a  year  in  advance,  typically,  of 
when  the  money  finally  reaches  the  university.  The  long  lead  time  between 
budget  preparation  and  receipt  of  money  also  makes  it  impossible  to  predict 
the  exact  costs  of  high  technology  items  such  as  computers.  You  know  that 
all  the  universities  have  federally  acceptable  procurement  procedures. 
When  the  time  comes  to  actually  order  equipment,  we  get  the  best  current 
price  through  competitive  bids. 

Increase  the  amount  that  can  be  spent  on  an  item  of  capital  equipment  that 
was  not  listed  in  the  original  proposal  from  $2,500  to  $5,000. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Stronger  commitment  by  NASA  to  fund  at  levels  indicated  in  the  NRA. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 


♦  I  also  would  suggest  that  page  limits  be  increased  slightly  to  allow  more  fig¬ 
ures,  that  is,  specifically  and  exclusively  figures.  I  do  not  believe  that  the 
length  of  text  should  be  increased,  because  review  panels  already  have  a  lot 
to  read.  However,  an  additional  page  or  two  of  figures  does  not  usually 
represent  nearly  as  much  work  to  read  as  a  page  of  text,  and  figures  are  im¬ 
portant  in  clarifying  the  proposal  and  in  supporting  its  points. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  description  of  appropriate  research  should  include  the  goals  and  initia¬ 
tives  [that]  NASA  sees  [for]  the  program.  For  example,  if  NASA  would  like 
to  see  studies  emphasizing  the  interaction  between  different  magnetospheric 
regions,  this  should  be  stated. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  A  system  should  be  introduced  to  pre-sort  the  competition.  Perhaps  an  ex¬ 
panded  letter  of  intent  (say  4  pages)  of  pre-proposal  material  should  be  in¬ 
vited,  evaluated,  and  then  full  proposals  invited  from  the  top  30  -  40%  only. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Make  the  NRA  instructions  less  bureaucratic  and  [more]  accurate.  In  one 
NRA  last  year,  manned  space  flight  opportunities  were  listed  as  a  possibility 
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in  the  NRA,  but  NASA/HQ  personnel  said  that  was  wrong.  Must  we  inter¬ 
rogate  HQ  to  verify  every  statement  in  an  NRA? 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Limit  text  of  proposal  to  10  pages. 

Reduce  forms  to  simple  cover  page.  Why  collect  certifications  up  front? 
Why  not  submit  them  only  upon  acceptance? 

Likewise,  until  proposal  [is]  actually  accepted,  why  not  limit  budget  to  sin¬ 
gle  page  with  minimum  "stats"  only? 

Isn't  there  some  why  to  determine  dollars  actually  available  at  the  time  the 
announcement  is  made? 

If  less  than  5-10  proposals  can  be  awarded  in  any  one  round,  dispense  with 
panel. 

Requiring  separate  proposals  from  co-investigators  at  other  institutions  is 
wasteful.  Could  do  that  paperwork  after  acceptance. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  playing  field  should  be  level  for  all  proposers.  Proposals  from  NASA 
centers  should  show  full  cost  of  research,  including  salaries  and  burdens.  At 
present,  NASA  proposals  show  an  artificially  low  cost.  Although  this  is  not 
supposed  to  be  a  factor  in  selection,  it  is. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Grant  requirements  are  spread  all  over  [the]  document.  Unify  [them]  in  one 
place. 

Give  a  sample  submission  (even  though  you  say  you  don't  like  required 
forms). 

The  boiler-plate  part  is  getting  awfully  hard  to  read. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  would  suggest  that  the  science  grant  review  process  should  include  a  step 
which  allowed  proposers  to  address  objections  raised  by  reviewers  before 
grant  selection  was  made.  That  is,  proposers  should  receive  a  copy  of  re¬ 
viewers  comments  before  the  review  panel  meets  and  should  be  allowed  to 
provide  written  responses  for  consideration  by  the  panel.  This  is  similar  to 
the  review  process  for  journal  publication,  and  the  entire  process  could  still 
be  accomplished  within  a  3-month  period. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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♦  The  NRAs  should  be  more  specific  for  each  solicitation  and  less  generic.  The 
current  format  is  not  straightforward  enough.  I  find  myself  constantly  re¬ 
reading  each  section  to  make  certain  that  the  particular  proposal  I  am  writ¬ 
ing  will  not  violate  some  obscure  limitation  in  one  section  or  another  of  the 
NRA.  I  think  the  current  page  limits  should  exclude  figures  since  too  much 
time  is  spent  reducing  them  in  size  and  number  to  fit  within  the  current 
page  limits.  I  agree  to  limits  on  text  length,  but  the  inclusion  of  graphics  and 
text  in  the  overall  length  of  the  Technical  Description  portion  of  the  proposal 
has  meant  that  each  proposal  is  now  like  a  camera-ready  manuscript  for  a 
volume  of  conference  proceedings.  Too  much  time  is  spent  trying  to  meet 
the  restrictions.  Even  as  a  reviewer  of  many  proposals,  I  would  rather  see 
more  of  what  the  proposers  have  accomplished  and  what  they  plan  to  do 
than  the  current  restrictions  allow.  Perhaps  allow  renewal  proposals  to  in¬ 
clude  less  repetitive  "boilerplate"  bios,  publications,  facilities,  etc.  would  al¬ 
low  those  proposals  to  be  shortened.  Allow  5-year  grants  to  cut  down  on 
frequency  of  proposals. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  NASA  should  pay  even  more  attention  to  the  broad  recommendations  of  its 
expert  review  panels. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  am  troubled  by  the  growing  set  of  "certifications"  that  must  be  added  to  a 
proposal  ("lobbying,"  etc.).  That  should  be  curtailed  or  eliminated  —  insti¬ 
tutions  should  be  able  to  acknowledge  Federal  rules  once  a  year  in  a  single 
letter. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Panel  reviews  were  given  poor  instructions  on  how  to  select  proposals.  In 
one  recent  case,  the  panel  chair  violated  the  direct  NRA  instructions,  and  the 
contractor  running  the  meeting  didn't  correct  this. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Suggestions:  (1)  Electronic  forms  for  proposals  as  the  instructions  now  are 
quite  involved.  (2)  It  would  be  a  great  improvement  if  more  outside  re¬ 
viewers  were  used  and  if  they  were  encouraged  to  be  fair  and  look  at  the 
larger  picture.  It  appears  to  me  that  programmatic  and  personal/institution 
connections  are  playing  much  too  large  a  role  in  some  very  larger  and  sig¬ 
nificant  decisions. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  In  the  proposal  announcements,  the  areas  of  interest  should  be  clearly  stated 
and  prioritized. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  NRAs  are  often  hard  to  understand. 
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The  scope  of  NRA  94-055-14  was  changed  by  a  "Dear  Colleague  Letter" 
dated  2  weeks  before  the  proposal  was  due.  I  received  this  letter  one  day  be¬ 
fore  the  due  date.  Although  this  change  did  not  affect  my  proposal,  the  tim¬ 
ing  of  this  announcement  indicates  a  lack  of  seriousness  on  the  part  of  the 
Program,  and  a  lack  of  respect  for  the  proposers. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Some  programs  are  closed  to  outsiders.  Please  fix  this:  Astrophysics  The¬ 
ory;  KAO;  High  Energy  (GRO,  etc.). 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Standardize  budget  forms  and  proposal  formats/ cover  pages,  etc. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Set  up  jointly  funded  programs  with  other  agencies  (e.g.,  NSF,  NOAA,  etc.) 
whereby  the  use  of  NASA  mission  data  to  solve  problems  relevant  to  these 
agencies  could  be  investigated  —  i.e.,  set  aside  funding  to  expand  the  utility 
of  NASA  data  and  products  to  other  agencies. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Rules  such  as  only  50%  of  the  grant  being  allowed  for  equipment  (in  some 
programs)  should  be  eliminated  or  waived  in  specific  cases. 

Above  all,  NASA  should  cap  allowable  indirect  costs  to:  40%  on  salaries 
and  wages  and  benefits;  15%  on  supplies,  travel,  publication  costs,  etc.;  0% 
on  equipment,  student  stipends  and  tuition. 

Also  in  the  case  of  the  PI  and  co-investigators  (CIs)  being  at  different  institu¬ 
tions,  a  mechanism  is  needed  to  prevent  unnecessary  indirect  costs  being 
levied  on  transfer  of  funds  to  the  Cl's  institution. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Use  a  [grant]  system  closer  to  that  of  Canada. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  A  process  of  appeal  for  unfair  reviews  would  be  a  great  help. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  NASA  should  be  clearer  what  "NASA  related"  means  for  theoretical 
proposers  —  both  to  the  proposer  and  to  evaluation  committees. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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♦  Why  the  need  for  detailed  budgets  on  a  grant  proposal,  when  90%  of  the 
time,  if  selected,  a  new  budget  is  required  (always,  less  money  than  pro¬ 
posed). 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  would  like  to  see  an  evaluation  option  for  each  of  the  specific  evaluation 
factors,  not  just  the  factors  all  together.  In  particular,  "Relevance"  is  an  al¬ 
most  worthless  evaluation  factor,  and  I  bet  most  of  the  community  feels  that 
way. 

General 

♦  Make  sure  that  proposals  are  evaluated  in  an  even-handed  way. 

►  In  the  past  NASA  has  played  favorites  and  caved  into  political  pressure 
to  change  awards. 

►  There  have  also  been  occasions  when  technical  objectives  changed 
while  proposals  were  in  evaluation. 

Some  of  these  factors  have  improved  of  late.  Keep  it  up. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Consider  [submitting]  the  proposals  anonymously  to  the  reviewers,  so  they 
are  judged  entirely  on  the  proposal  merits  rather  than  on  the  reputation  of 
the  PI. 

Obtain  reviews  of  proposals  from  professionals  who  (a)  have  time  to  do  a 
thorough  review  and  (b)  understand  the  specific  subfield.  Many  of  my 
negative  reviews  come  from  people  who  obviously  have  no  idea  of  the  type 
of  work  I  am  proposing  and  who  spend  little  time  reading  the  proposal  it¬ 
self. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  am  particularly  troubled  [by  the  following:]  Of  primary  concern  in  evalua¬ 
tion  and  fund  allocation  is  whether  the  applicant  is  an  "astronomer"  —  not 
whether  the  problem  of  substantial  interest  to  astronomy  merits  support.  I 
have  been  advised  to  arrange  collaboration  with  persons  in  our  astronomy 
department  prior  to  submitting  a  proposal. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Feedback  must  be  [logical.]  Lately,  the  comments  (when  rejected)  appear  to 
reflect  a  desire  to  cull  proposals  for  suspect  reasons  rather  than  the  result  of 
a  considered  judgment. 
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♦  There  are  artificial  "walls"  between  Planetary  Astronomy  and  Planetary  Geol¬ 
ogy  and  Geophyics.  Sometimes  there  are  projects  that  can  shift  in  emphasis 
from  one  program  to  the  other  on  a  time-scale  [of]  approximately  3  years. 
At  present,  each  "shift"  puts  the  proposal  into  the  category  of  a  "new  PI" 
making  the  chances  for  success  much  lower,  even  for  an  outstanding  pro¬ 
posal.  Work  of  this  type  is  infeasible  given  the  present  funding  system. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Provide  unedited  copies  of  reviews  to  Pis  (the  way  NSF  does);  if  that  is  un¬ 
acceptable  to  NASA,  then  at  least  provide  Pis  fuller  information  regarding 
the  reviews. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Evaluation  is  extremely  flawed.  Emphasis  is  on  data  processing,  [but]  not 
much  thought  is  given  to  work  which  could  be  of  long-range  value.  Very 
disappointing  at  the  present. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Grants  are  often  funded  at  reduced  level  [which  is]  inadequate  for  [meeting] 
project  goals. 

Fund  grants  fully,  even  if  it  means  fewer  total  grants. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Some  of  my  proposals  have  gotten  great  feedback  and  reviews,  some  only  a 
few  paragraphs,  and  some  nothing  at  all.  The  few  in  which  I  have  gotten  no 
feedback  were  principally  instrument  upgrade  proposals  for  which  I  heard 
later  that  no  funding  was  available,  but  it  would  still  be  nice  to  get  some 
comments. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Some  statistical  data  from  previous  calls  for  proposals  (e.g.,  #  proposals,  # 
accepted)  would  help  me  to  decide  whether  it's  worth  responding  to  a  par¬ 
ticular  NRA. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Specific  written  comments  justifying  the  denial  of  a  proposal  by  the  NASA 
administrator  responsible  for  selection  are  essential  feedback  to  guarantee 
fairness  in  the  grant  process.  The  step  between  review  panel  and  final  selec¬ 
tion  seems  very  mystical! 

More  information  concerning  those  proposals  such  as,  making  them  pub¬ 
licly  available  since  they  have  now  become  public  contracts,  would  be 
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valuable  to  the  entire  community  and  would  lead  to  improved  NASA  pro¬ 
grams  in  the  long  run. 

Generally,  more  openness  on  the  part  of  NASA  is  needed. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Far  too  much  effort  and  cost  are  spent  on  the  reference  process  for  only  a 
10  -  20%  rate  of  funding.  A  lottery  would  be  more  efficient  and  just  as  fair. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  have  participated  on  both  sides  of  the  grant  review  process  —  having  sub¬ 
mitted  proposals  for  review  and  serving  on  NASA  grant  review  panels.  I 
strongly  believe  that  the  panel  review  process  as  currently  structured  is  in¬ 
adequate.  My  foremost  concern  is  that  a  large  number  of  proposals  are  not 
exposed  to  a  sufficient  number  of  reviewers  who  have,  the  necessary  exper¬ 
tise  in  the  specific  field  of  inquiry.  Very  often,  the  panels  are  too  small, 
especially  in  some  programs  which  have  a  tremendous  scope  (e.g..  Astro¬ 
physics  Theory).  In  addition,  I  feel  that  more  use  of  mail-in  reviews  should 
be  made.  I  am  aware,  for  example,  of  some  reviews  in  which  NO  outside  re¬ 
views  were  solicited.  Furthermore,  in  many  instances  when  mail-in  reviews 
were  used,  the  vast  majority  (if  not  all)  of  reviews  are  from  persons  submit¬ 
ting  concurrent  proposals.  While  this  should  not  necessarily  disqualify  a  re¬ 
viewer,  the  emphasis  should  be  to  select  mail-in  reviewers  on  the  basis  of 
competence  and  expertise  in  the  area.  In  short,  a  carefully  selected  mail-in 
panel  is  necessary  to  counter  possible  deficiencies  in  the  general  panel. 

NASA  feedback  in  the  form  of  a  panel  "consensus"  is,  at  best,  inadequate  . . . 
Too  often,  the  final  reviews  are  watered  down  and  do  not  reflect  the  full 
range  of  issues  which  were  brought  up  in  the  panel  discussions.  I  believe 
that  in  addition  to  a  more  detailed  panel  report  (which  should  be  closer  to  a 
discussion  summary  than  a  "consensus"),  the  investigators  should  be  sent 
the  complete  mail-in  reviews,  as  well  as  the  initial  reports  of  the  principal 
and  secondary  reviewers. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  By  and  large,  I  think  NASA  does  a  good  job  of  evaluating  and  handling 
grant  proposals  in  a  timely  manner.  Evaluation  by  a  small  committee  is  su¬ 
perior  to  evaluation  by  isolated  individuals. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  NASA  is  one  of  the  best  agencies  in  handling,  evaluating,  and  granting  pro¬ 
posals.  Although  I  think  things  could  be  better,  NASA  is  "light  years" 
ahead  of  agencies  like  the  USGS,  EPA,  DOE  and  others. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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♦  The  comments  returned  from  review  are  totally  inadequate.  Often  they  are 
terse.  They  generally  look  like  they  were  just  put  there  to  match  [the]  deci¬ 
sion  on  that  particular  grant. 

Breaking  into  a  new  program  is  difficult. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Peer  review  panels  should  be  eliminated.  If  panels  are  "here  to  stay"  then 
the  proposer  should  have  an  opportunity  to  respond  to  the  mail-in  reviews 
before  the  panel  meets. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Please  make  sure  that  your  program  managers  are  qualified  to  judge  the 
topics  proposed!  A  Ph.D.  should  be  a  requirement  for  ALL  program  manag¬ 
ers.  (Shows  dedication  to  research  and  scientific  understanding.) 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  process  is  important,  but  the  main  problem  is  that  there  is  too  little 
money  in  the  programs  to  meet  the  needs.  Also,  the  individual  grants  are 
too  small  in  many  cases,  forcing  one  individual  to  write  3  to  5  proposals  to 
support  a  single  program. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  think  proposers  should  automatically  receive  all  "frontside  copies"  of  all 
external  and  panel  reviews.  This  is  very  valuable  feedback.  It  would  also 
force  reviewers  to  be  more  careful  and  responsible.  I  sometimes  suspect  that 
some  external  reviewers  are  overly  hasty,  possibly  petty,  etc.  This  can  be  fil¬ 
tered  out  in  the  panel.  However,  often  useful  detailed  feedback  by  genuine 
experts  is  also  filtered  out  and  lost  in  the  "summary"  reviews.  Very  often 
this  is  the  only  way  truly  honest  critiques  and  feedback  are  obtained. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  When  reviewing  proposals  for  scientific  merit,  do  not  include  the  investiga¬ 
tors'  names  or  institutions.  I  do  believe  that  there  is  a  great  bias  towards 
those  from  more  well-known  institutions.  I  have  experienced  this  when  I 
moved  from  an  institution  more  well-known  than  where  I  am  now.  My  sci¬ 
ence  efforts  have  not  changed,  but  I  have  noticed  an  increase  in  difficulty  in 
being  awarded  grants. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  [Hire]  more  competent  NASA  program  officers. 
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♦  Generally,  I  think  the  system  works  well  and  almost  as  fairly  as  is  possible 
when  using  the  peer-review  process.  Better  feedback,  especially  more  con¬ 
structive  criticism,  would  help  with  rejected  proposals. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  would  like  to  see  the  name  of  the  proposer  kept  anonymous  to  all  the  re¬ 
viewers. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  disagree  with  the  idea  of  asking  proposers  to  review  proposals.  I  believe 
this  is  an  obvious  conflict  of  interest  which  benefits  no  one. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Send  postcards  acknowledging  receipt  of  proposals  within  2  weeks.  Send 
"blind"  copies  of  reviewers  comments  to  Pis,  NOT  summaries  by  discipline 
scientist! 


♦  In  many  cases  it  was  obvious  from  the  (limited)  feedback  that  proposal  re¬ 
viewing  was  not  done  properly.  In  many  cases,  it  was  obvious  that  the  peer 
review  was  actively  looking  for  "excuses"  not  to  fund  the  proposal. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Although  I  have  been  rather  successful  in  my  dealings  with  NASA,  I  do  not 
feel  that  the  proposal  selection  process  is  strictly  fair.  The  selection  process 
should  be  re-examined  for  objectivity.  Reviewers  should  have  absolutely  no 
vested  interest  in  the  outcome  of  the  review  process. 

- ♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  entire  process,  from  drafting  the  NRA  to  making  the  final  decision  on 
which  grants  to  fund,  takes  about  a  year.  By  the  time  the  panels  convene 
and  the  director  makes  the  final  decisions,  a  significant  part  of  the 
scientific/programmatic  priorities  set  forth  in  the  NRA  have  undergone  at 
least  one,  if  not  more  revisions.  Since  we  cannot  possibly  ask  the  panel  re¬ 
viewers  and/or  the  program  director  to  make  choices  against  the  current 
scientific/programmatic  priorities  for  the  sole  purpose  of  sticking  to  the 
NRA,  the  best  solution  is  to  shorten  the  length  of  the  process,  preferably  to 
under  six  months  (from  drafting  the  NRA  to  the  final  decision).  That  way 
most  of  the  information  on  the  NRA  will  still  be  relevant  by  the  time  the  fi¬ 
nal  decision  is  made.  It  is  very  frustrating  to  make  painstaking  efforts  to 
keep  one's  proposal  within  the  programmatic  priorities  set  in  the  NRA,  only 
to  discover  that  the  reviewers  rated  as  scientifically  more  relevant  methods 
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and/ or  lines  of  investigation  that  were  left  out  of  the  proposal  because  they 
were  not  mentioned  in  the  NRA. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  very  strongly  support  the  panel  review  process  which  NASA  currently 
uses  to  evaluate  proposals.  In  my  opinion,  this  process  is  much  fairer  (and 
therefore  more  effective)  than  the  reliance  on  remote  peer-reviewers  used  by 
NSF,  for  example.  The  simultaneous  review  of  a  group  of  proposals,  and 
the  discussions  among  reviewers,  lead  to  better  decisions.  Keep  up  the  good 
work! 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  You  need  to  infiltrate  as  many  college  campuses  as  you  can,  as  many  depart¬ 
ments  as  you  can,  if  you  want  to  develop  more  of  a  collegial  "atmosphere." 
It's  almost  as  if  "government  science"  is  different  than  "academic  science." 
Give  a  lot  of  little  grants  to  graduate  students  through  faculty  members.  It 
would  have  a  potential  impact  on  the  space  knowledge  base. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  My  dissatisfaction  is  mainly  driven  by  a  problem  that  is  beyond  NASA's  di¬ 
rect  control:  the  current  proposal  system  is  an  enormous  waste  of  time.  It 
simply  defies  all  logic  to  imagine  that  time  and  money  are  being  used  effi¬ 
ciently  when  one  must  submit  5-10  proposals  for  each  one  granted. 

I  have  received  "negative  comments"  on  proposals  I  have  submitted:  (a)  for 
not  citing  papers  that  were  not  yet  published  when  the  proposal  was  sub¬ 
mitted,  and  (b)  that  demonstrated  explicit  ignorance  of  the  work  in  the  field 
by  the  referees.  I  am  fully  aware  that  proposal  refereeing  is  both  difficult 
and  tedious.  That  does  excuse  either  the  referee  nor  NASA  for  shoddy 
work. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Maintain  a  healthier  theory  program.  Don't  overload  it  by  stripping  all  the¬ 
ory  elements  out  of  other  programs. 

Provide  support  for  small-scale  (i.e.,  individual  investigator)  grants.  It's  un¬ 
healthy  to  have  a  few,  very  big  groups  get  all  the  funding  in  this  area. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  very  system  of  proposal  writing  should  be  drastically  changed.  Too 
many  scientists  are  wasting  their  precious  time  to  write  too  many  proposals 
every  year  —  and,  in  most  cases,  cannot  cover  even  their  own  salary,  let 
alone  hiring  an  assistant,  even  if  funds  are  granted  in  full.  Other  scientists 
have  to  read /review  a  lot  of  proposals  which,  in  turn,  takes  a  toll  on  their 
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time  and  productivity. 


Also,  NASA's  science  grant  process  leaves  too  little  or  no  chance  at  all  for 
young  scientists  to  get  into  any  program.  Competition  is  high  and  the  ab¬ 
sence  of  just  one  right  "keyword"  gives  a  low  score  to  the  whole  proposal. 
Newcomers  are  at  disadvantage  when  they  compete  with  old  experienced 
Pis. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  most  serious  issue  regards  the  lack  of  coherence  in  the  grant  process  in 
the  space  physics  community. 

In  the  distant  past,  a  research  grant  was  awarded  for  many  years  and  for 
support  of  many  researchers  located  at  one  institution  working  coherently 
on  a  project.  Many  researchers,  including  graduate  students,  were  funded 
under  the  auspices  of  this  grant,  and  with  the  coherent  effort,  NASA  was 
getting  a  lot  of  scientific  "bang  for  the  buck."  Specifically,  ideas  spawned 
within  the  group  made  for  thesis  topics  and  papers  for  the  younger  scien¬ 
tists.  The  "think  tank"  atmosphere  of  the  group  allowed  for  free  thought 
and  an  uninterrupted  development  of  ideas.  At  HQ,  the  disbursement  of 
funds  was  much  easier.  One  large  proposal  justifying  the  existence  of  the 
group  was  required,  rather  than  a  large  set  of  small  proposals  to  justify  indi¬ 
vidual,  specific  topics.  The  most  important  factor  is  that  the  worthiness  of 
the  individual  topics  were  decided  at  the  institution  level  and  not  at  HQ 
level.  Thus,  large  HQ  review  committees  and  the  like  were  not  really  re¬ 
quired  to  verify  topic- worthiness.  It  was  all  transparent  to  HQ. 

Unfortunately,  not  every  institution  could  obtain  a  large  multi-year,  group 
proposal. 

In  the  mid-80s,  an  effort  was  made  to  create  some  parity.  In  the  process,  the 
coherent  groups  were  disassociated.  Research  grants  were  given  on  a  topic- 
by-topic  basis,  which  gave  money  based  on  ideas  rather  than  on  the  institu¬ 
tion.  In  theory  this  concept  seems  correct.  In  reality,  researchers  are  now 
writing  as  many  as  5  proposals  a  year  to  piecemeal  their  funding.  The  large, 
coherent  "think  tanks"  are  gone.  What  little  money  is  left  is  dispersed,  but 
no  one  is  talking.  HQ  has  to  sponsor  these  elaborate  review  committees  to 
study  each  proposal,  which  is  now  written  on  a  topic-by-topic  basis.  The 
loss  of  coherency  means  that  NASA  is  not  getting  the  most  for  its  dollar. 
And  individual  researchers  (particularly  on  soft  money)  now  spend  all  their 
time  writing  proposals  rather  than  doing  research.  The  system  is  now  fair 
and  honest,  but  not  very  efficient. 

The  solution:  Set  up  a  hybrid  process.  Identify  specific  institutions  as 
"think  tank"  centers  and  guarantee  funding  based  on  the  worthiness  of  the 
group.  However,  save  some  money  for  individual  topics  that  can  be  re¬ 
viewed  by  the  mail-in  process.  Eliminate  the  review  by  committee  system. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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♦  Some  way  should  be  found  to  reduce  the  review  costs  and  use  those  savings 
to  fund  more  grants.  It  is  my  opinion  that  any  method  used  will  not  be  both 
cost-effective  and  fair  until  the  number  of  grants  is  [approximately] 
30  -  35  percent  of  the  number  of  proposals.  The  proposed  solution  is  ideal¬ 
ized  because  I  realize  that  it  is  not  simple  to  use  proposal  review  money 
from  one  NASA  money  pot  for  grants  that  come  from  another  NASA  money 
pot.  However,  the  unfairness  in  the  selection  process  can  be  eliminated  only 
by  instituting  a  new  review  process  and  increasing  the  funding  of  the  over¬ 
all  program. 

My  specific  changes  are:  (1)  Scrap  the  review  panel  process  (it  is  much  too 
costly).  (2)  Obtain  the  services  of  a  small  group  of  individuals  who  will  ex¬ 
tract  meaningful  write-in  reviews  from  the  community.  (This  is  a  difficult 
task  given  that  many  in  the  community  do  not  take  the  time  to  review  pro¬ 
posals,  but  it  is  certainly  more  cost  effective  than  the  current  system.) 
(3)  Convene  a  small  panel  of  scientists  to  sift  through  those  proposals  that 
fall  within  the  top  30  to  40  percent  of  the  write-in  reviews  and  get  their  rec¬ 
ommendation  on  ONLY  these  proposals. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  key  is  having  a  knowledgeable,  proactive  Discipline  Scientist  for  each 
program.  Unfortunately,  NASA  HQ  has  filled  the  DS  functions  by  cycling 
inexperienced  people  through  such  positions  at  2  -  3  year  intervals.  The  re¬ 
sult  has  been  insufficient  continuity  and  oversight. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  NASA  should  adopt  the  NSF  mode  of  operation.  Proposals  may  be  solicited 
or  unsolicited  but  should  be  taken  throughout  the  year.  The  once  a  year 
deadline  and  the  expensive  yearly  panel  review  process  should  be  abol¬ 
ished.  The  money  saved  should  be  diverted  to  research  activities.  This  will 
also  reduce  the  peak  load  once  a  year  and  make  the  contractor  service  re¬ 
dundant,  leading  to  further  savings  which  can  be  used  for  research.  The 
program  manager  should  be  given  more  authority  to  judge  the  proposal  on 
scientific  merits  and  programmatic  needs.  Mail-in  review  may  be  used  for 
guidance  but  should  not  be  binding.  A  procedure  to  rebut  and  resubmit 
proposals  should  be  instituted.  This  will  reduce  conflict  of  interest  prob¬ 
lems  and  give  the  proposer  a  fair  opportunity  to  establish  the  validity  of 
his/her  ideas. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Prefer  independent  individual  reviews  over  panel  [reviews].  Panel  [is]  sub¬ 
ject  to  politicking  by  one  dominant  member. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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♦ 


I  am  very  satisfied  with  the  overall  aspects  of  grant  submission,  review,  and 
feedback. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 


♦  Programmatic  considerations  must  be  made  open;  otherwise  the  process  gives 
advantage  to  some  groups. 

A  division  director  should  personally  debrief  a  PI  if  a  proposal  is  rated  as 
"excellent"  but  does  not  receive  funding. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  After  the  grant  is  awarded  and  the  appropriate  documents  are  transferred 
from  the  technical  monitor's  office  to  the  office  which  handles  financial  mat¬ 
ters,  the  grant  invariably  gets  "lost."  Numerous  phone  calls  from  us  are  re¬ 
quired  in  order  to  receive  payment.  We  are  often  told  that  the  grant  papers 
are  "lost"  in  huge  stacks  of  paperwork  on  various  individual's  desks.  Once 
the  initial  payment  is  received,  it  can  take  multiple  faxes  to  receive  later  pay¬ 
ments.  Why  can't  this  be  handled  over  the  Internet?  NSF  has  developed  an 
excellent  system  in  this  regard  and  it  is  quite  efficient. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  In  the  "Science"  subgroups  of  NASA  funding,  the  review  process  becomes 
too  much  of  a  "good-old-boys"  process,  with  little  chance  of  outsider  entry. 
A  critical  review  of  the  allocations  by  some  educated  outside  group,  about 
every  5  years,  would  be  very  beneficial. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  If  would  be  helpful  to  receive  copies  of  the  reviewers'/ panel  members  com¬ 
ments  on  the  proposal.  Also,  it  is  very  difficult  to  reach  people  at  the  pro¬ 
gram  director  level  to  discuss  ideas  for  proposal  research  and  receive 
guidance. 

The  bureaucracy  governing  NASA  funding  appears  to  be  immense,  and  it's 
not  at  all  clear  what  role  scientific  merit  plays  in  the  selection  of  successful 
proposals. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  My  chief  complaint  is  that  funding  for  analysis  of  specific  data  sets  often 
comes  in  increments  that  are  small  (in  time  and  dollars).  This  makes  it  quite 
difficult  to  use  this  money  effectively  to  support  graduate  student  or  even 
post-doctorate  [researchers].  It  is  much  more  efficient  to  fund  in  amounts 
and  over  time  intervals  that  allow  [continuous]  support  [of  the]  research 
program.  Overall,  more  funding  of  programs  is  desirable  (even  at  the  ex¬ 
pense  of  more  specific  "mission-oriented"  funding). 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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♦  A  greater  effort  should  be  made  to  see  to  it  that  all  proposals  are  referred  by 
at  least  3  independent  referees.  All  referee  comments  should  be  sent  to  the  in¬ 
vestigators,  not  just  a  committee  summary.  Referees  should  be  asked  to, 
whenever  possible,  suggest  ways  that  the  investigators  could  improve  their 
proposals.  When  evaluation  and  selection  programs,  including  the  names  of 
the  investigators,  research  topic,  period  of  funding  and  level  of  funding 
should  be  sent  to  everyone  who  submitted  a  proposal.  Proposal  abstracts 
should  also  be  readily  available.  The  process  should,  in  general,  be  more 
open. 

The  Discipline  Scientist  at  NASA  Headquarters  should  be  familiar  and  com¬ 
fortable  with  all  areas  of  the  discipline.  He  or  she  should  also  be  familiar 
with  all  submitted  proposals  (and  should  have  sufficient  time  to  do  this). 
The  DS  should  have  significant  input  into  the  selection  process  (based  upon 
scientific  merit  and  relevance  to  NASA  projects)  and  should  not  simply  de¬ 
termine,  based  upon  a  point  system,  where  the  funding  cutoff  will  be. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  think  the  method  of  evaluating  proposals  in  review  committee/panels  has 
some  weaknesses,  but  the  advantage  of  having  it  done  in  such  a  timely  fash¬ 
ion  is  a  strong  advantage.  I  support  continued  use  of  review  panels  to  keep 
the  feedback  time  short. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  recommend  that  oversubscribed  NASA  proposals  for  theory  and  modeling 
be  awarded  on  a  lottery  basis.  For  the  sake  of  argument,  my  definition  of 
oversubscribed  is  that  less  than  20%  of  submitted  proposals  will  be  funded. 
Under  the  present  system,  NASA  must  evaluate  and  rank  all  proposals.  The 
methods  used  are  mail-in  reviews,  panel  discussions,  and  a  combination  of 
the  two.  I  would  estimate  that  80%  of  all  proposals  submitted  are  worthy 
investigations  and  would  be  funded  if  resources  were  available.  The 
evaluation  and  ranking  process  is  therefore  a  difficult  one.  Although  the 
present  methods  appear  fair  and  reasonable,  they  are  more  of  a  facade  than 
a  meaningful  discriminator.  It  is  not  that  peer  reviewers  are  incompetent 
but  that  their  job  is  too  difficult.  Relatively  minor  details  are  being  used  to 
differentiate  proposals;  this  is  not  valid  for  a  scientific  research  project.  Pis 
are  given  a  wide  latitude  in  conducting  basic  research,  often  following  un¬ 
foreseen  paths  that  develop  as  new  research  results  are  obtain.  They  cannot 
be  held  accountable  for  every  minor  aspect  of  a  proposal.  After  all,  NASA 
does  not  have  program  reviews  in  which  Pis  are  required  to  demonstrate 
that  they  are  carrying  out  research  objectives  exactly  as  specified  in  their 
proposal.  Thus,  it  seems  fairer  to  simply  have  a  lottery  system.  NASA  can 
cull  out  the  truly  bad  proposals  using  mail-in  reviews  and  programmatic 
judgment.  The  remaining  proposals  "go  into  a  hat"  and  the  winners  are 
pulled  out.  Aside  from  being  fairer,  this  method  would  save  NASA  the 
money  spent  on  the  present  review  process;  this  money  could  be  used  to 
fund  additional  programs. 
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Proposal  costs  should  only  be  evaluated  by  NASA  and  not  scientific  review¬ 
ers.  The  costs  associated  with  the  government,  universities,  and  private 
companies  are  different  (e.g.,  overhead)  and  comparing  them  with  each 
other  is  comparing  "apples  to  oranges."  NASA  management  is  in  a  better 
position  to  make  the  evaluation  since  it  is  familiar  with  the  various  cost  dif¬ 
ferences;  in  general,  individual  research  scientists  are  not  in  such  a  position. 
Also,  detailed  travel  cost  information  (e.g.,  plane  fare,  hotel  costs,  etc.)  is  not 
needed.  First,  it  is  a  waste  of  time  for  a  scientist  to  busy  him/herself  with 
such  minutiae,  and  second  does  NASA  ever  follow  up  and  check  to  see  how 
research  funds  are  spent  on  travel? 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  One  [criticism]  is  the  lack  of  feedback  in  research  areas  that  are  hardware- 
oriented.  By  this  I  mean,  groups  or  investigators  whose  hardware  is  gener¬ 
ally  unsuccessful  apparently  suffer  no  penalties  when  proposing  for  new 
programs.  New  missions  and  studies  seem  to  be  awarded  to  whomever 
puts  together  the  most  sweeping  claims  for  new  science  investigations,  and 
although  NASA's  programs  should  certainly  be  science-driven,  technical 
merit  —  as  evidenced  by  successful  hardware  operations  —  should  also  be 
rewarded.  Review  panels  should  be  made  fully  aware  of  each  proposer's 
track  record,  as  it  were,  and  this  information  given  reasonable  weighting  in 
allocating  resources.  This  is  a  difficult  task.  In  a  small  community,  many  of 
the  most  knowledgeable  potential  reviewers  may  be  unable  to  serve  on  the 
panel,  as  they  are  likely  to  submit  their  own  proposals.  Furthermore,  there 
is  a  tendency  for  errors  and  omissions  to  be  swept  under  the  rug,  perhaps  to 
avoid  embarrassing  the  group  or  NASA  more  generally. 

The  second  [criticism]  is  NASA's  schizophrenia  about  "pre-selecting"  mis¬ 
sions  or  supporting  missions  in  study  phases.  A  successful  proposal  for  a 
mission  of  any  significant  scope  (UNEX,  SMEX,  MIDEX,  etc.)  requires  an 
enormous  amount  of  resources.  NASA  may  provide  only  a  few  months  be¬ 
tween  announcements  and  proposal  due  date,  but  the  old-fashioned  notion 
that  a  professor  and  a  student  or  two  can  sit  down  and  write  a  successful 
proposal  in  the  "spare  time"  available  during  the  few  month  period  is  hope¬ 
lessly  outdated.  Many  science  investigators  are  often  required,  and,  more 
importantly,  a  large  investment  of  resources  —  good  old  fashioned 
money  —  is  needed.  Engineers  must  be  paid,  junior  scientists  must  carry 
out  detailed  simulations,  schedules  and  budgets  must  be  developed  by  man¬ 
agers.  All  of  this  involves  travel,  time,  and  MONEY.  An  investigator  who, 
for  whatever  reason,  has  current  resources  at  his/her  disposal,  has  an  enor¬ 
mous  advantage  over  another  investigator  who  does  not. 

Sometimes  these  resources  are  provided  by  his/her  academic  or  industrial 
institution.  This  of  course  is  fine;  institutions  that  support  proposal  prepara¬ 
tion  merit  an  advantage  for  their  "investment."  More  often,  though,  the  re¬ 
sources  are  not  provided  by  the  institution,  but  are  borrowed,  by  "hook  or 
by  crook,"  from  existing  NASA  projects.  There  is  no  way  to  stop  this  from 
happening. 
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To  address  this  instability,  one  would  hope  that  NASA  could  find  ways  to 
distribute  funding  to  promising  investigators  to  enable  them  to  prepare 
competitive  proposals,  to  level  the  playing  field.  But  herein  is  the  schizo¬ 
phrenia.  Funding  officials  react  in  (mock)  horror  to  such  a  suggestion. 
"You  want  NASA  to  pre-select  missions  before  proposals  are  formally  due? 
What  would  the  lawyers  say!"  The  irony,  of  course,  is  that  NASA,  currently, 
pre-selects  proposals  form  well-funded  groups,  with  no  scientific  peer  re¬ 
view  whatsoever. 

The  solution  is  to  structure  certain  announcements  explicitly  to  provide 
funding  for  mission  studies.  The  recent  "Mission  Concepts"  announcements 
is  a  step  in  the  right  direction.  However,  the  announcements  was  flawed  for 
several  reasons.  First,  only  those  instruments  which  are  not  currently  tech¬ 
nically  feasible  would  seem  to  fall  in  this  category.  Second,  despite  a  clear 
statement  in  the  announcements  that  missions  of  a  variety  of  scopes  were 
solicited,  rumor  subsequently  emerged  that  only  very  large,  multi¬ 
investigator,  multi-hundred-million-dollar  missions  were  truly  desired. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  More  emphasis  on  science  quality  with  lesser  emphasis  on  programmatic  as¬ 
pects  would  improve  the  impact  and  quality  of  research  in  the  long  run. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  There  is  only  one  thing  to  improve.  NASA  usually  has  little  money  for  new 
proposals  in  each  period.  Therefore:  (a)  lengthen  period;  (b)  simplify  re¬ 
viewing  process  (too  many  reviewers;  too  much  worry  about  fairness;  it's 
o.k.  if  NASA  has  lots  of  money  to  give  out).  Rely  more  on  contract  monitors 
and  outside  reviews  —  "kill"  the  expensive  panels. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Feedback  is  extremely  important.  However,  for  legal  reasons  NASA  cur¬ 
rently  limits  the  proposer  to  a  panel  summary  statement.  It  would  be  most 
helpful  to  be  given  the  front  pages  of  the  actual  reviews.  Less  would  be  lost 
in  the  translation.  It  is  also  important  to  have  knowledgeable  and  experi¬ 
enced  discipline  scientists.  This  is  not  always  the  case. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  All  my  comments  are  based  on  what  NASA  HQ  does  with  the  resources 
available.  Like  many  scientists  of  late,  I  have  received  marks  of  excellent 
and  very  good  on  proposals  that  could  not  be  funded  because  of  resource 
limitations.  That  is,  of  course,  a  different  problem  from  managing  fairly 
with  constrained  resources.  There  I  think  HQ  is  doing  well. 
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♦  I  am  very  concerned  that  there  is  a  tendency  to  keep  as  many  researchers  as 
possible  alive  in  an  epoch  of  decreasing  budgets.  The  method  of  spreading 
the  pain  often  has  an  adverse  impact  on  research  productivity.  Develop¬ 
ment  efforts  take  significant  sums  of  money  over  long  periods  of  time. 
When  the  funding  level  falls  below  critical  levels,  many  investigators  do  lit¬ 
tle  more  than  mark  time  and  put  the  best  face  they  can  on  their  limited  pro¬ 
gress.  Others  find  themselves  in  a  mode  of  writing  numerous  proposals  to 
maintain  economies  of  scale.  Researchers  often  write  proposals  that  request 
a  level  of  funding  they  think  can  be  awarded  rather  than  an  amount  they  ac¬ 
tually  think  they  need.  Then  these  researchers  have  to  contend  with  funding 
officers  restricting  their  budgets  further.  The  end  result  is:  the  fraction  of 
time  spent  on  proposal-writing  activities  is  too  large  compared  to  the  re¬ 
search  returned. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  problem  with  the  system  is  that  it  makes  us  devote  our  effort  to  prepar¬ 
ing  high-ranked  proposals,  rather  than  doing  good  science.  New  ideas  are 
usually  badly  rated  because  they  have  not  been  proven  to  work;  doing  old 
ideas  over  again  a  little  better  is  necessary  to  be  funded.  We  are  all  forced  to 
play  "musical  chairs"  for  grant  support,  instead  of  trying  to  do  good  science 
(risky  —  it  sometimes  fails). 

Solutions:  1.  Spread  the  money  thinly  enough  that  every  productive  scien¬ 
tist  has  a  little  (at  least  enough  to  go  to  a  meeting,  perhaps  even  summer  sal¬ 
ary),  and  can  stop  worrying  about  his  next  proposal  and  start  thinking  about 
science  again.  This  continual  worry!  What  will  I  propose  next?  Will  it  suc¬ 
ceed?  Drains  our  creative  and  productive  energy. 

2.  Restrict  each  PI  to  one  government  grant  at  a  time.  Once  you've  got  one, 
it's  time  to  think  about  science,  not  proposals!  This  would  also  encourage 
institutions  to  give  young  people  independence  (so  they  can  be  Pis).  It 
would  reduce  the  number  of  proposals  written  (which  NASA  must  review) 
by  a  large  factor. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  By  and  large  there  is  no  problem  with  the  system  for  evaluating  proposals. 
The  problem  is  the  shortage  of  money  for  distribution.  There  are  dozens  of 
proposals  that  are  rated  excellent,  scoring  1  on  a  1  to  5  scale,  and  are  de¬ 
clined  funding. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  proposal  process  should  be  greatly  simplified,  with  much  greater  em¬ 
phasis  placed  on  the  Pi's  track  record.  Ideally,  the  process  should  resemble 
what  I  understand  has  been  the  Canadian  system,  whereby  the  proposal 
consists  only  of  a  general  outline  of  proposed  research  and  is  judged  to  a 
large  extent  on  past  and  ongoing  performance.  A  related  simplification 
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would  be  achieved  with  guaranteed  funding  in  each  cycle  for  5  years  or  so 
subject  to  periodic  performance  review. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  believe  that  the  number  of  excellent  proposals  greatly  exceeds  the  number 
that  can  be  funded,  and  that  the  process  of  selecting  those  which  are  funded 
therefore  involves  subjective  judgments,  [which  can]  be  quite  arbitrary.  I 
suggest  that  all  proposals  which  exceed  some  objective  selection  threshold 
be  treated  equally  and  selected  by  lottery.  External  reviews  should  suffice 
for  ranking  the  proposals,  and  review  panels  will  not  be  needed.  I  believe 
that  such  a  system  would  be  simpler  and  more  fair.  Proposers  should  have 
an  opportunity  to  respond  to  unfair  reviews. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  suggest  that  the  review  process  have  two  phases,  with  the  first  phase  send¬ 
ing  comments  and  criticism  to  the  proposer,  who  could  then  respond  in  the 
second  phase.  It  often  happens  that  a  review  committee  is  misinformed,  and 
this  would  correct  that  problem. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  It  is  important  that  NASA  seriously  consider  supporting  qualified  investiga¬ 
tors  from  schools  other  than  the  "big  name"  schools.  Those  investigators 
have  fresh  ideas  and  new  technologies.  They  are  eager  to  [contribute  to]  the 
knowledge  sought  by  NASA.  They  cost  less  and  [in]  a  lot  of  cases  [are]  bet¬ 
ter  equipped. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  grant  process  is  very  fair  and  democratic,  it  just  needs  more  funding  to 
keep  American  Space  Science  alive. 

-♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Comments  from  [the]  review  panel  should  be  more  specific  and  detailed. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  scientists  who  review  a  proposal  should  not  be  those  whose  research  in¬ 
terests  are  in  competition  (or  even  the  rivals  of)  the  proposed  research. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Average  award  sizes  must  be  increased,  even  if  this  means  fewer  awards. 
Small  awards  force  Pis  to  submit  multiple  proposals,  which  is  generating  ex¬ 
cessive  work  throughout  the  entire  process. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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♦  The  research  programs  that  NASA  has  set  up,  and  the  way  those  programs 
are  run  and  proposals  reviewed,  are  excellent.  But,  most  programs  are  un¬ 
derfunded,  and  the  NASA  infrastructure  to  handle  these  programs  is 
dwindling  to  the  point  that  unconscionable  delays  in  review/ feedback/ fund 
ing  are  now  occurring.  The  whole  system  is  sinking  under  its  own  weight. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Panel  review  should  be  eliminated. 

Great  care  should  be  taken  in  the  selection  of  reviewers  to  ensure  compe¬ 
tence  and  minimize  conflict  of  interest.  Suggestions  for  potential  reviewers 
from  proposers  should  be  encouraged. 

Full  disclosure  of  all  reviews  and  reasons  for  funding  or  not  funding  should 
be  automatically  provided  with  the  notification  of  outcome. 

There  is  too  much  emphasis  on  "guaranteed  results"  projects.  More  atten¬ 
tion  should  be  directed  to  new  "risky"  science. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  In  the  past  2  to  3  years,  NASA  Program  Managers  have  not  been  providing 
extensive  review  comments  on  proposals.  This  is  a  major  method  of  im¬ 
proving  one's  science,  since  those  comments  are  often  valuable.  I  suggest 
that  if  proposers  sign  a  waiver  stating  their  intention  not  to  sue  NASA  they 
should  be  shown  anonymous  review  comments. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  wonder  why  I  rank  the  review  process  highly  but  am  not  as  satisfied  with 
the  overall  science  grant  process?  That's  because  I  have  to  write  too  many 
proposals  to  raise  enough  money  to  continue  my  research.  This  is  a  tough 
field  to  be  in.  The  survey  also  makes  me  realize  that  I'm  not  as  bad  off  as  I 
thought  and  also  that  things  have  changed  in  the  past  years.  Proposal  an¬ 
nouncements  are  better  than  they  used  to  be;  I  still  hate  costing. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  More  complete  feedback. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Discourage  excessive  advisory  committees.  They  waste  funds  on  travel  and 
tend  to  be  populated  by  persons  who  either  stand  to  gain  directly  from  the 
advice  given,  or  are  "professional"  committee  members  whose  own  research 
has  languished.  NASA  would  benefit  from  a  complete  dismantling  of  the 
entire  subculture  of  advisory  boards  and  committees.  Program  directors 
and  discipline  scientists  should  take  more  responsibility  for  leadership. 
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There  are  also  too  many  NRAs.  Each  one  gets  wide  distribution,  creating  a 
blizzard  of  proposals  from  people  who  try  to  put  a  spin  on  their  own  ongo¬ 
ing  research  so  that  it  fits  the  NRA.  Such  proposals  tend  to  be  either  redun¬ 
dant  or  half-baked,  putting  an  unnecessary  burden  on  review  panels  and 
tending  to  reduce  average  grant  size. 

The  peer  review  process  is  generally  conscientious  and  ethical,  much  im¬ 
proved  from  14  years  ago  when  I  started  proposing  to  NASA.  There  are  two 
weaknesses:  too  small  a  pool  of  mail-in  reviewers,  and  not  enough  instru¬ 
mentalists  among  mail-ins  or  panels.  The  excuse  given  for  the  small  number 
of  mail-in  reviewers  is  that  every  reviewer  must  have  signed  a  non¬ 
disclosure  form  before  receiving  a  proposal.  I  would  fix  that  by  simply  not 
accepting  proposals  containing  proprietary  information. 

It  is  not  fair  for  NASA  centers  to  compete  directly  with  universities;  the  cen¬ 
ters  should  be  encouraged  to  complement  rather  than  duplicate  university 
research. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Be  more  supportive  of  ground-based  research  when  it  impacts  the  results  of 
space  missions.  Provide  more  detailed  feedback  on  why  a  proposal  is  re¬ 
jected. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  [Provide]  more  elaborate  feedback  on  proposals  not  approved. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  NASA  peer-review  process,  with  panels  evaluating  many  proposals  si¬ 
multaneously,  seems  excellent  to  me;  well  worth  the  cost. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 


♦  The  one  aspect  of  the  proposal  process  with  which  I  am  most  dissatisfied  is 
that  it  sometimes  appears  that  the  reviewer(s)  did  not  read  the  proposal 
carefully  enough.  It  is  one  thing  when  a  reviewer  does  not  agree  with  an  ar¬ 
gument  in  the  proposal;  it  is  another  matter  when  a  reviewer  erroneously 
claims  that  a  subject  or  problem  was  not  addressed  at  all.  This  problem  is 
no  doubt  due  to  the  large  number  of  proposals  which  must  be  reviewed.  I 
urge  that  NASA  increase  the  size  of  review  panels  so  that  proposals  can  re¬ 
ceive  more  careful  scrutiny. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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♦  Evaluation  process  too  heavily  biased  in  favor  of  experimenters  in  receipt  of 
existing  funds.  New  operators  find  it  difficult  to  "break-in"  to  this  "grants 
for  the  boys"  type  of  situation. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  More  consistency  in  feedback  [is  needed].  Sometimes  it  comes,  sometimes  it 
doesn't.  Sometimes  it  is  objective,  sometimes  it  isn't. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  have  found  the  feedback,  i.e.,  review  and  critique  of  proposals,  to  be  unin¬ 
formative.  The  lack  of  adequate  feedback  as  to  what  was  good  and  bad 
about  a  proposal  is  very  disturbing.  [Inadequate  feedback  has  been]  re¬ 
peated  on  every  proposal  I  have  submitted,  both  those  that  have  been 
funded  as  well  as  those  that  have  not  been  funded. 

I  suggested  that  the  process  used  by  the  National  Science  Foundation  be 
used.  When  a  proposer  receives  the  letter  stating  that  the  proposal  will  or 
will  not  be  funded,  the  actual  written  comments  of  the  individual  reviewers 
are  included  in  addition  to  the  panel  review  summary. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Remove  the  "old  boy"  network  that  is  pervasive  throughout  NASA's  grant 
funding  process.  Not  that  all  NASA  funded  science  is  laughable,  but  NASA 
should,  through  the  program  director  (NOT  the  committee  selected  to  evalu¬ 
ate  proposals),  keep  a  tighter  control  in  the  quality  of  science  being  funded. 
This  requires  qualified  and  experienced  people  to  be  program  directors.  In 
my  experience,  this  is  not  the  case! 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  There  seems  [to  be]  no  legitimate  reason  for  HQ  to  return  only  a  summary  of 
proposal  reviewers  comments.  Return  [of]  all  reviewer  comments  would  ex¬ 
pedite  the  feedback  process. 

I  am  worried  about  the  proposal  review  panel  composition  in  terms  of  con¬ 
flict  of  interest  and  the  ability  of  some  colleagues  to  be  objective  in  their 
judgment. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Many  announcements  are  without  funding  and  just  seem  to  disappear. 

A  common  opinion  exists  in  the  community  that  the  "old  boys"  network  is 
alive  and  well,  and  most  grants  are  "hard  wired"  or  at  least  heavily  biased 
to  present  researchers  —  perceived  as  hard  to  break  in,  even  with  good  ideas. 
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Selected  individuals  (old  boys?)  seem  to  know  more  of  the  specifics,  not 
broadly  known. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  More  effort  needed  to  ensure  impartial  review  panels. 

More  effort  needed  to  ensure  competing  proposals  are  evaluated  to  a  com¬ 
mon  standard. 

Since  so  few  proposals  are  currently  being  funded,  some  effort  needs  to  be 
made  to  identify  priority  areas  in  the  announcements.  To  reduce  the  number 
of  man-hours  wasted  on  proposals  that  have  little  chance  of  success  pro¬ 
grammatically  (i.e.,  some  programmatic  decisions  should  be  made  ahead  of 
time). 

Encourage  the  American  people/ congress  to  provide  for  a  stable  budget  so 
that  long  term  plans  can  be  made  and  kept  to. 

Discourage  use  of  funds  to  support  students  (and  post-doctorate  research¬ 
ers?)  until  such  time  as  the  downsizing  of  the  community  enforced  by  cur¬ 
rent  budgetary  constraints  has  been  achieved. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Reinvent  the  NRA  process  so  that  everybody  is  assumed  to  get  $0K  at  the 
start  of  their  proposal  period.  If  they  submit  a  poor  proposal,  they  get  $0K, 
if  they  submit  a  good  proposal,  they  get  what  they  ask  for.  This  will  remove 
the  significant  advantage  that  incumbent  proposers  now  enjoy,  and  reduce 
the  stagnation  and  waste. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  [Reduce  the  number  of]  blanket  NRAs.  More  specific  calls  would  keep  peo¬ 
ple  from  submitting  proposals  to  every  NRA  reminiscent  of  their  research 
area. 

Startup  grants  for  new  researchers  is  basically  absent  from  NASA  grants. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  approach  taken  of  rotating  a  large  number  of  investigators  through  the 
panels  ensures  as  fair  a  process  as  possible.  It  should  be  continued. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  NASA  division  staff  is  excellent,  works  very  hard  and  is  sensible  in  its 
decisions.  The  problem  is  that  they  are  understaffed  and  unable  to  keep  up 
with  the  many  programs  they  have  to  manage. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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♦  (1)  The  major  improvements  in  the  science  grant  process  should  be:  a.) 
support  of  the  fundamental  science,  methods  for  theoretical  analyses;  b.)  fair 
evaluation:  concrete  discussion  of  the  reasons  for  rejecting  a  proposal  and 
supporting  a  proposal;  a  possibility  to  give  a  rebuttal  in  the  case  when  the 
reasons  for  rejecting  a  proposal  are  wrong,  c.)  a  feedback  from  NASA  which 
indicates  clearly  whether  NASA  is  satisfied  with  the  research  of  the  Princi¬ 
pal  Investigator  or  not,  and  by  what  reasons,  d.)  at  the  end  of  the  research 
period  (grant  period)  there  should  be  a  final  evaluation  of  the  research  com¬ 
pleted,  and  a  detailed  discussion  of  the  strong  and  weak  points  in  the  com¬ 
pleted  research.  Good  work  should  be  explicitly  recognized  as  well  as  a  bad 
[work.]  (2)  A  toll-free  number  for  communication  with  NASA  should  be 
available. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  Peer  Review  panels  generally  are  good,  although  I  have  seen  some  indi¬ 
cation  of  the  "good  old  boys"  network  trying  to  "kill"  some  fields  or  prevent 
newcomers  from  getting  grants. 

Some  grants  have  become  entitlements  with  Pis  getting  big  sums  even  after 
their  contributions  have  decreased.  This  is  bad  [because]  a  fairly  small  frac¬ 
tion  of  the  Pis  get  a  large  fraction  of  the  funds. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Its  generally  a  good  process,  but  at  times  it  appears  exceedingly  difficult  for 
new  research  ideas  and  people  to  break  into  an  area. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  have  served  as  an  outside  peer  reviewer  of  NASA  proposals  in  3  different 
programs  and  I  have  served  on  the  panel  of  scientists  which  reviews  propos¬ 
als  in  the  light  of  outside  reviews.  I  think  this  process  is  extremely  fair  and 
objective  especially  in  contrast  to  other  agencies  with  which  I  am  familiar. 

♦  Make  more  use  of  electronic  media  (WWW).  Make  statistics  available  on 
how  program  money  is  spent  (administrative  cost,  review  panel  cost,  etc.). 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  More  money  [is]  spent  writing  proposals  than  was  distributed.  This  is  idi¬ 
ocy  beyond  words.  NASA  is  in  deep  trouble  in  its  grants  program.  It  has 
become  a  real  lottery.  The  merit  system  is  DEAD  in  the  grants  program! 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Money  should  be  earmarked  to  fund  grants  for  a  post-doctural  researcher  at 
non-government  labs  (i.e.,  a  counterpart  to  the  NRC  fellowship  program). 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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♦  The  general  problem  that  I  perceive  is  a  gradual  deterioration,  over  the  past 
decade  or  more,  in  the  adequacy  of  the  funding  level  to  complete  a  given 
program.  The  effect  of  this  is  to  drive  the  researcher  into  producing  more 
proposals  to  maintain  a  coherent  program  and  to  spend  more  [time  writing 
proposals  than  doing  the  research].  The  effect  of  this  is  to  increase  the 
Headquarters  overhead  in  dealing  with  an  increasing  flood  of  proposals,  re¬ 
ducing  the  efficiency  of  the  overall  program. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Provide  modest  funding  for  the  review  teams,  and  —  if  NASA  is  too  busy  to 
handle  this  paperwork  themselves  —  make  the  teams  responsible  for  provid¬ 
ing  (at  least)  the  scientific  feedback  to  proposers. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  most  important  thing  I  would  like  to  see  improved  is  better  feedback. 
The  summary  of  strengths  and  weaknesses  from  a  proposal  evaluation  are 
given  in  a  form  so  terse  as  [to]  be  almost  useless.  Comments  sometime  al¬ 
most  seem  contradictory.  A  few  sentences  would  be  more  helpful  than  3  or 
4  two  word  phases. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  realize  that,  with  today's  budgets,  the  majority  of  NRA  proposals  are  going 
to  lose,  but  as  someone  who  has  participated  in  an  NRA  review  committee 
meeting,  it  seems  that  an  enormous  amount  of  money  is  spent  just  in  the  re¬ 
view  process  alone.  If  NRAs  were  processed  perhaps  every  other  year  (or 
every  third  year)  would  this  not  help  reduce  this  cost?  I  am  sure  you  are 
thinking  of  ways  to  streamline  this  process  and  the  sooner  this  is  done  (if  it 
is  possible)  the  better!  I  know  it  is  a  difficult  job  and  you  are  doing  the  best 
you  can.  Most  of  the  Space  Scientists  I  know  are  also  doing  the  best  they 
can.  Hopefully  we  can  all  work  together  in  the  future  and  make  space  ex¬ 
ploration  and  space  science  exciting  and  important. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  In  some  areas  NASA  relies  on  a  relatively  small  number  of  experts  which 
leads  to  a  "skewed"  review  process.  The  "Old  Boys"  network  should  be 
avoided.  I  also  should  add  that  the  person  responsible  for  feedback  should 
be  more  forthcoming.  Also  "one  liner"  reports  should  be  avoided.  After  all, 
one  spends  a  lot  of  time  submitting  a  proposal.  One  is  therefore  entitled  to  a 
decent,  extensive  report. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  NASA  feedback  on  the  proposals  seem  to  vary  across  the  different  divi¬ 
sions  and  subdisciplines.  In  Space  Physics  and  Astrophysics  divisions  the 
panel  consensus  reports  are  provided  along  with  the  NASA  decision  on  the 


J-39 


proposal.  In  the  past  the  Planetary  Atmospheres  Program  of  the  Solar  Sys¬ 
tem  Exploration  Division  provided  the  reports  only  over  the  telephone, 
which  is  a  tedious  process.  NASA  should  adopt  uniform  procedures  for 
providing  its  feedback. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Panel  reviews  have  had  a  lot  of  variations  in  their  quality.  Some  were  hast¬ 
ily  chosen  and  were  staffed  with  people  10  years  "out  of  date"  in  their  re¬ 
search  familiarity. 

What  to  do?  Longer  lead  times  in  selecting  panelists,  more  care  in  selecting 
high-quality  active  researchers  to  serve  on  the  panels.  With  active  recusal 
policies  for  individuals  with  conflicts  of  interest,  the  panels  might  be  able  to 
be  improved  in  quality  by  using  people  more  familiar  with  the  spacecraft 
and  fields  of  research. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  [It  should  be]  mandatory  that  NRAs  contain  enough  information  to  let  us 
know  what  the  scope  and  capabilities  of  the  mission  are  all  about.  It  takes  a 
lot  of  time  to  get  documents  over  the  network  and  print  them  out,  just  to  see 
if  the  announcement  is  of  any  interest.  If  you  want  to  make  such  informa¬ 
tion  available  over  the  WWW,  that's  fine  too. 

May  I  point  out  that  in  the  old  days  the  paper  NRAs  served  a  very  useful 
purpose?  They  made  sure  that  you  people  at  HQ  and  out  at  the  centers 
were  organized:  all  of  the  user /proposer  information  had  to  be  ready  for  the 
printers.  Now,  with  electronic  distribution  of  the  NRAs,  the  information  in 
the  files  is  often  incomplete  or  even  wrong!  You  cannot  expect  to  be  chang¬ 
ing  this  documentation  two  weeks  before  the  proposal  deadline,  as  has  hap¬ 
pened  recently.  This  just  makes  the  process  too  chaotic  and  too 
time-consuming,  and  it  also  jeopardizes  the  legality  of  the  whole  review. 
Someone,  someday,  may  become  disgruntled  enough  to  challenge  the  whole 
electronic  process  unless  you  take  care  to  ensure  it  is  done  as  orderly  as  if  it 
had  been  done  "on  paper"  the  way  NASA  did  it  in  years  past. 

In  peer  review,  NASA  is  starting  to  look  real  bad,  if  you  ask  me.  There  was 
a  time  when  the  peer  reviews  were  done  astonishingly  well.  Now  the  pan¬ 
els  tend  to  be  too  small  to  have  sufficient  expertise  to  cover  the  broad  areas 
of  research  they've  been  assigned,  and  the  members  of  the  panels  are  given 
too  heavy  a  load  of  proposals  to  review.  It  is  simply  not  possible  to  have  a 
fair  and  thoughtful  review  if  a  panelist  is  assigned  60  to  70  proposals,  and  if 
the  formal  meetings  are  limited  to  at  best  2  days.  I  have  been  told  that  at  the 
last  ASCA  review,  many  panelists  had  not  even  read  the  proposals  when 
they  met.  Where's  the  fairness  in  that? 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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♦  [Overall]  the  process  is  reasonably  good  and,  more  importantly,  fair.  How¬ 
ever,  I  do  believe  there  are  areas  [in  need  of]  improvement. 

(1)  Be  specific  about  the  areas  for  funding. 

(2)  2-phase  process  (similar  to  NIH).  1st  stage:  short  2  page  summary  of 
proposed  work  etc.  Only  have  full  proposals  for  final  stage.  This  saves  Pis 
time  and  money. 

(3)  Be  more  specific  in  debriefings.  Help  the  Pis  to  produce  better  proposals 
next  time.  Again,  NIH  provides  feedback  to  encourage  proposers  who  did 
well  (but  not  funded)  for  the  next  round. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Discipline  program  managers  [should]  not  stay  in  their  offices  for  more  than 
a  3-year  term. 

Panel  review  system  is  sometimes  quite  unfair  and  should  be  replaced  by  a 
mail-in  peer  review  process. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Individual  researcher  and  small  group  efforts  are  vitally  important  for  get¬ 
ting  the  science  out.  I  hope  NASA  will  continue  to  support  them  as  well  as 
they  have. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Don't  change  the  selection  rules  from  those  of  the  NRA.  If  there  is  to  be  so¬ 
cial  engineering,  announce  it,  so  people  don't  waste  their  lives. 

Encourage  reviewers  to  accept  some  longer  proposals. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  feel  the  proposal  evaluations  have  been  uninformed  and  capricious.  I 
think  I  would  prefer  the  mail-in  review,  as  in  the  NSF  AIM  program  — 
even  though  it  may  take  longer.  The  panel  process  outcome  seems  to  de¬ 
pend  on  the  judgment  of  one  person,  who  may  not  be  informed. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Established  programs  should  make  sure  that  some  money  is  allocated  to 
new  starts.  Otherwise,  the  process  becomes  closed  and  resistant  to  new 
ideas.  It  is  difficult  enough  as  it  is  to  justify  a  new  idea  compared  with  on¬ 
going  established  research  programs. 

Writing  a  full  proposal  every  year  to  fund  a  $30K  grant  is  not  a  productive 
use  of  NASA's  grant  money.  I  could  be  doing  research  instead! 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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♦  I  strongly  recommend  changing  the  review  process  to  that  followed  by 
NSF  —  with  mail-in  reviews  rather  than  committee  reviews. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  suggest  strongly  that  junior  colleagues,  with  lack  of  experience  of  referee¬ 
ing  business,  NOT  be  asked  to  serve  in  the  peer  review  panels.  They  are 
simply  NOT  PEERS!  They  do  not  do  that  in  other  countries,  England,  Japan, 
Germany,  etc. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  (1)  Written  feedback  suggests  that  proposals  are  sometimes  not  understood 
or  not  read  carefully  (or  at  all).  I  have  also  seen  a  poor  level  of  preparation 
for  proposal  review  meetings.  (2)  NASA  is  using  too  many  junior  review¬ 
ers,  or  people  with  a  poor  overall  understanding.  It  is  not  correct  to  use 
"post-docs"  to  review  proposals:  they  have  inadequate  experience  and 
breadth.  (3)  I  believe  that  proposal  review  is  "fair"  —  in  the  sense  that  sci¬ 
ence  is  the  main  criterion.  (4)  Remove  "current  and  pending"  page  from 
proposals  sent  out  for  review.  Many  reviewers  are  envious  of  those  with 
other  well  funded  programs.  (5)  I  think  that  NASA  is  wrong  to  earmark 
specific  amounts  of  money  for  junior  investigators  (e.g.,  LTSA 
program  —  2/3  money  for  juniors).  We  already  have  too  many  astronomers 
and  this  practice  will  exacerbate  personnel  problems.  There  are  not  enough 
good  juniors  to  justify  such  generous  funding.  (6)  Drastically  reduce  num¬ 
ber  of  small  and  short-term  grants  and  increase  funding  to  LTSA.  The 
short/small  programs  are  extremely  inefficient  of  everyone's  time.  The 
same  people  are  applying  for  both  small  and  LTSA  programs. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Please  continue  to  keep  the  budgeting  simple! 

While  project  officers  do  a  very  good  job,  after  the  process  goes  beyond  their 
control  it  bogs  down  in  terms  of  the  amount  of  time  between  arrival  of  the 
award  letter  and  the  eventual  arrival  of  the  contract/ check.  In  one  case  it 
took  16  months! 

Keep  communication  lines  (such  as  this  questionnaire)  open! 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 


♦  Return  copies  of  all  review  comments  to  proposers.  Do  so  promptly  upon 
completion  of  reviews.  Have  formal  appeal  process  for  bad  reviews.  Have 
appeal  process  for  inadequate  funding.  Provide  more  checks  [to  ensure] 
that  [the]  review  process  is  fair  and  unbiased. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 


J-42 


♦  My  experience  has  been  that  an  "outsider"  has  little  or  no  chance  of  having  a 
research  proposal  funded  by  NASA. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  have  been  involved  with  Space  Physics  as  a  proposer,  PI  and  member  of  re¬ 
view  panels.  Review  panel  process  needs  improvement.  Not  all  panel 
members  see  the  proposal.  Too  often,  a  proposal  rating  is  pushed  up  or 
down  by  only  one  or  two  panel  members.  Sometimes,  personal  bias  seems 
to  play  a  role,  especially  with  dominant,  senior  people  on  that  panel.  Feed¬ 
back  to  proposers  could  be  expanded  —  currently  only  a  few  terse  lines.  I 
think  that  NASA  (Space  Physics)  is  really  trying  to  be  fair  and  objective.  It  is 
an  overwhelming  job. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  I  applaud  (in  general)  the  panel  review  process.  However,  I  believe  that  the 
process  is  biased  against  innovative  research.  This  is  based  on  personal  ex¬ 
perience  as  a  proposer  and  as  a  panel  member. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Since  75%  of  grant  money  goes  to  only  a  few  institutions,  why  not  just  give 
5-year  block  grants  and  let  the  lesser  institutions  fight  for  the  remaining  25% 
funds  via  the  NRA's.  Top  institutions  could  be  peer  reviewed  every  5  years. 
This  would  save  a  huge  amount  of  time  and  effort  in  proposal  writing. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  In  many  cases  those  who  organize  and  run  the  review  committees  are  also 
the  same  people  who  run  the  actual  (data  center  satellite)  and,  naturally, 
write  a  significant  number  of  proposals.  This  inevitably  leads  to  conflicts  of 
interest  which  have,  on  occasion,  led  to  heavily  biased  proposal  selections. 

1. )  I  strongly  urge  that  procedures  be  changed  to  avoid  such  conflicts  of  interest.  A 
separate,  neutral  organization  should  deal  with  the  complete  review  process 
from  committee  selection  to  running  the  review  to  distributing  the  reports. 
An  excellent  example  here  is  running  of  the  NASA  ADP  program  review 
process. 

2. )  Strict  rules  of  decorum  should  be  applied  during  the  process.  For  exam¬ 
ple,  technical  people  should  be  available  for  consultation  but  not  present  in 
the  committee  room(s)  at  any  other  time. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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♦  NASA's  successes  have  been  truly  outstanding.  So,  there  must  be  many 
good  qualities  in  its  science  grant  process.  However,  I  have  two  problems 
with  NASA's  science  grant  process: 

(1)  I  feel  the  process  is  a  camaraderie  system,  at  least  in  part.  What  do  I 
mean  by  that?  It  is  inevitable  that  during  periods  of  austerity  and  re¬ 
assessment  such  as  NASA  is  currently  going  through  that  Agency  personnel 
who  are  making  programmatic  decisions  need  help.  What  must  they  do 
when  they  do  not  have  the  funds  to  pay  for  scientific  expertise?  They  "bor¬ 
row"  it  by  inviting  scientists  from  the  universities  and  research  laboratories 
to  form  working  groups  and  so  forth.  The  said  university  scientist  gains 
from  this  symbiosis  by  having  an  advantage  when  RFPs  or  NRAs  go  out. 
The  problem  is  that  as  human  beings,  there  is  a  natural  tendency  for  the 
peer-reviewer  (specifically,  the  Headquarters  Division  involved)  to  feel  that 
"something"  (grants,  funds,  or  money)  is  owed  back  to  these  people  who 
helped.  Inevitably,  peer  review  creates  a  clique  of  "you-cite-me-and-I-  will- 
cite-you"  mentality,  with  reviewers  no  longer  "at  arms  length"  from  the  re¬ 
viewed  as  they  should  be. 

(2)  Presently,  NASA's  system  is  sort  of  a  "blind  bid"  system  for  the  pro¬ 
poser  (unless  the  PI  is  "in"  the  camaraderie  group).  This  is  backwards  and 
is  inequitable.  What  do  I  mean?  If  you  are  "in"  the  system,  you  can  get  the 
grant.  If  you  fail,  the  only  feedback  you  get  is  a  cryptic  critique.  So,  you  are 
basically  "blind"  to  what  it  takes  to  win  the  grant  and  to  what  is  going  on 
(what  NASA  "wants").  How  do  you  do  something  about  this?  (2a)  Provide 
a  fair  and  equitable  feedback  system  wherein  the  total  results  of  the  grant 
process  are  made  public  knowledge  to  all  proposers,  including  the  losers. 
This  would  at  least  include  titles  of  the  successful  proposals  and  what  the 
size  of  the  grant  was.  (2b)  A  better  feedback  process  will  necessarily  im¬ 
prove  the  quality  of  subsequent  grants  from  the  losers  —  and  by  definition 
that  is  what  the  feedback  should  be:  Whatever  improves  the  quality  of  the 
NASA  science  grant  system.  (2c)  What  is  the  correlation  between  NASA's 
lofty  research  goals  and  research  actually  funded  to  pursue  those  goals? 
Publish  this  correlation  in  the  acceptance/ rejection  cycle.  (2d)  Is  NASA  on  a 
kite-flying  exercise?  If  you  do  not  want  speculative  proposals,  then  NASA 
should  not  be  speculating  about  the  origin  of  the  Universe.  (2e)  Draw  a  line 
in  the  sand  between  NSF  research  and  NASA  research,  because  one  of  the 
oldest  tricks  in  the  book  is  to  tell  a  new  investigator  that  his/her  proposal 
should  [go]  to  the  NSF  and  not  NASA.  (2f)  Stop  advertising  goals  if  and 
when  you  have  no  intention  of  funding  them.  Otherwise,  you  create  a  credi¬ 
bility  problem  with  the  science  community  when  you  invite  the  naive  PI 
into  a  response. 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Having  participated  in  all  aspects  of  the  NASA  grant  process  over  the  years, 
I  believe  that  many  problems  as  perceived  by  the  community  at  this  time 
rest  in  the  general  inadequacy  of  funding.  Many  highly  meritorious  propos¬ 
als  end  up  not  funded.  The  peer  review  process  is  very  good  in  separating 
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meritorious  from  non-meritorious  proposals,  but  it  is  much  less  successful 
in  distinguishing  between  similarly  meritorious  proposals  other  than  on  the 
basis  of  minor,  mostly  irrelevant  factors.  This  gives  rise  to  a  great  amount  of 
frustration  among  proposers.  Any  amount  of  efficiency  in  managing  the 
process  is  not  going  to  change  that. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  Feedback  [is]  generally  not  very  informative.  I  believe  it  is  important  for  the 
review  panels  to  provide  meaningful  comments  for  the  deficiencies  of  a  pro¬ 
posal.  So  the  proposer,  especially  if  he/she  is  a  junior  scientist,  can  receive 
useful  guidance  on  how  to  improve  their  proposal(s)  in  the  future.  Also,  the 
length  of  time  between  the  announcement  of  proposal  acceptance  and  fund¬ 
ing  start  should  be  shortened. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦  The  basic  problem  is  fairness.  1)  The  squeeze  put  on  research  funds  has 
probably  doubled  the  number  of  proposals  for  any  given  call.  One  way  to 
decrease  the  number  is  to  be  much  more  specific  in  the  call  as  to  total  money 
available,  total  number  of  grants  and  size  of  grants  and  an  explicit  break¬ 
down  to  the  areas  of  research  expected  to  be  funded  and  percent  of  monies 
which  will  support  that  work.  This  will  inject  a  little  bit  of  realism  into  the 
process  and  scientist's  expectations.  2)  The  review  committee's  make-up 
has  introduced  potential  for  bias.  Relevance  to  the  research  needs  of  com¬ 
mittee  members  appears  to  be  the  dominant  factor  affecting  scores,  not 
merit. 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
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